How would you encode your input vector/matrix from a sequence of moves in game like tasks to train an AI? e.g. Chess AI? I've seen data sets for classification / regressions tasks in domains such as credit default detection, object identification in an image, stock price prediction etc.  All of these data sets could simply be represented as an input matrix of size (n_samples, n_features) and fed into your machine learning algorithm to ultimately yield a trained model offering some predictive capability.  Intuitively and mathematically this makes sense to me.
However, I'm really struggling with how to think about the structure of an input matrix for game-like tasks (Chess, Go, Seth Blings Mario Kart AI) specifically (using the Chess example):

How would you encode the state of the board to something that a model could train on?  Is it reasonable to think about the board state as a 8x8 matrix (or 1x64) vector with each point being encoded by a numerical value dependent on the type of piece and color?

Assuming a suitable representation of the board state, how would the model be capable of making a recommendation given that each piece type moves differently?  Would it not have to evaluate the different move possibilities for each piece and propose which move it "thinks" would have the best long term outcome for the game?

A follow up on 2 - given the interplay between a moves made now and moves made n moves into the future how would the model be able to recognize and make trade-offs between moves which may offer a better position now vs those that offer a position n moves in the future - would one have to extend the board state input to a vector of length 1x64n where n is the total number of moves for expected for an individual player or is this a function of a different algorithm which should be able to capture historical information which training?

I am unsure if I'm overthinking this and am missing something really obvious but I would appreciate any guidance in terms of how to approach thinking about this.

The core of the question seems to really be: "how to approach thinking about this", where "this" is the input of an AI player.

Modern attempts at game playing AI players try to replace a human player "as is". No advantage whatsoever. This implies that we want to feed the same "raw input" to the software player and to the human player. For a video game, the raw input is usually the screen we see, but it could also include sound, vibration feedback from a game controller, force feedback from driving gears, etc. Basically any sensor input available. For a physical board game like Go, the raw input could be a video feed, as done for AlphaGo. Doing so, we fall back to the video game approach. So the input to the AI player is often a tensor, where each element is the intensity of a screen pixel, or some other sensor signal (note there is nothing here about the shape of the tensor, please see below).

The way of thinking here is actually pretty simple, and neat: We want an AI player to replace a human player. So we list up what the human player can sense, and model each input as a tensor (this includes vectors and matrices).

There is perhaps a subtlety here. We face choices not so obvious to make, and there are usually trials and errors (many errors). For example, the Deep-Q Network from DeepMind to play Atari games takes as input differences between two consecutive video frames. Some other approach could solve the same problem with actual sequences of inputs (e.g. 10 consecutive frames). The two approaches are valid---they just have different tradeoffs we must evaluate to find the best configuration.

Another subtelty is the "shape" of the input. Should a single (W, H) input frame be a (W*H) vector, or a (W, H) matrix, or something else? The ramifications can go far. If the frame is an RGB frame with transparency, the input could well be a tensor of shape (W, H, 4), a "cubic volume" with 4 "slices" of shape (W, H)---one for each RGB channel, and a fourth one for transparency. Imagine now if you also have sound, and can add a fifth "slice"---and more if you have stereo or Dolby system channels.

Last point: There is no reference to the architecture of the AI player beyond the input here. Although most exemples refer to recent architectures based on neural networks, this "thinking" also applies whatever can leverage the input. For example, the Big Blue system from IBM, who won Chess against Gary Kasparov, is an AI player using exclusively (if my memory's good) clever tree search. Yet its input is a tensor representing the Chess board.


Can a machine learning model predict the pattern of given sequence? I am curious if it is possible to do so.
For example, if I supply

$[0, 1, 2, 3, 4, 5]$, the model should return "natural number sequence",

$[1,3,5,7,9,11]$, it should return "natural number with step of $2$",

$[1,1,2,3,5]$, it should return "Fibonacci numbers",

$[1,4,9,16,25]$, it should return "square natural number"

and so on.

Those all fit into a single quadratic, auto-correlated model.

$$ x_0 = a \\ x_i = b i^2 + c x_{i-1} + d i + e $$

The sequences can be curve fitted producing a set of $n$ perfect fits of the form $(a, b, c, d, e)$ given the above model. A rules engine given the correct parameterized rules can produce the most desirable verbal description from among the $n$ fits in the set. The rules can also be prioritized by a simple feed forward network trained to simulate the most natural selection of string descriptions from any set of fits where $n > 1$.

This will work well for the examples in the question and many more, however, if the sequence $\{1, 4, 1, 5, 9\}$ is fed into the system, it will produce some weird description based on the quadratic, auto-correlated model it was given rather than, "digits of $\pi$ to the right of the decimal place."

The only way to produce the most common response a university freshman math student would produce would be to extend the boundaries of AI engineering first. For example, once an AI system is developed that can handle natural language and cognition like a child, several of them can be separately trained in a simulation of primary and secondary school mathematics. The median response for each sequence given to the class of AI students (class made up of artificial students studying math, not class of humans studying AI) will then be a reasonable prediction of what human university freshmen would produce as a median response.


In GradCAM, why is activation strength considered an indicator of relevant regions? In the GradCAM paper section 3 they implicitly propose that two things are needed to understand which areas of an input image contribute most to the output class (in a multi-label classification problem). That is:

$A^k$ the final feature maps
$\alpha_k^c$ the average pooled partial derivatives of the output class scores $y^c$ with respect to the the final feature maps $A_k$.

The second point is clear to me. The stronger the derivative, the more important the $k$th channel of the final feature maps is.
The first point is not, because the implicit assumption is that non-zero activations have more significance than activations close to zero. I know it's tempting to take that as a given, but for me it's not so obvious. After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means. We can easily transform two neurons [0, 1] to [1, 0] with a linear transformation.
So why should it matter which regions of the final feature maps are strongly activated?

EDIT
To address a comment further down, this table explains why I'm thinking about magnitude rather than sign of the activations.

It comes from thinking about the possible variations of
$$
L_{Grad-CAM}^c = ReLU\bigl( \sum_k \alpha_k^c A^k \bigr)
$$

I think you are misreading the relevant passage here.
Since you do not specify exact excerpt(s), I take that by "implicit assumption" you refer to the equation (2) (application of a ReLU) and the corresponding text explanation (bold emphasis mine):

We apply a ReLU to the linear combination of maps because we are only
interested in the features that have a positive influence on the class
of interest, i.e. pixels whose intensity should be increased in order
to increase $y^c$. Negative pixels are likely to belong to other
categories in the image. As expected, without this ReLU, localization maps sometimes highlight more than just the desired class and perform worse at localization.

The first thing to notice here is that this choice is not at all about activations close to zero, as you seem to believe,  but about negative ones; and since negative activations are indeed likely to belong to other categories/classes than the one being "explained" at a given trial, it is very natural to exclude them using a ReLU.
Grad-CAM maps are essentially localization ones; this is apparent already from the paper abstract (emphasis mine):

Our approach – Gradient-weighted Class Activation Mapping (Grad-CAM), uses the gradients of any target concept (say ‘dog’  in  a classification  network  or  a  sequence  of  words in captioning network) flowing into the final convolutional layer to produce a coarse localization map highlighting the important regions in the image for predicting the concept.

and they are even occasionally referred to as "Grad-CAM localizations" (e.g. in the caption of Fig. 14); taking a standard example figure from the paper, e.g. this part of Fig. 1:

it is hard to see how including the negative values of the map (i.e. removing the thresholding imposed by the ReLU) would not lead to maps that include irrelevant parts of the image, hence resulting in a worse localization.

A general remark: while your claim that

After all, neurons have biases, and a bias can arbitrarily shift the reference point, and hence what 0 means

is correct as long as we treat the network as an arbitrary mathematical model, we can no longer treat a trained network as such. For a trained network (which Grad-CAM is all about), the exact values of both biases & weights matter, and we cannot transform them arbitrarily.

UPDATE (after comments):

Are you pointing out that it's called "localization" and therefore must be so?

It is called "localization" because it is localization, literally ("look, here is the "dog" in the picture, not there").

I could make a similar challenge "why does positive mean X and why does negative mean Y, and why will this always be true in any trained network?"

It is not at all like that; positive means X and negative means not-X in the presence of class X (i.e. a specific X is present), in the specific network, and all this in a localization context; notice that Grad-CAM for "dog" is different from the one for "cat" in the picture above.

Why is it that a trained network tends to make 0 mean "insignficant" in the deeper layers? [...] why is it that a network should invariably make the positive activations be the ones that support the prediction rather than the negative ones? Couldn't a network just learn it the other way around but use a negative sign on the weights of the final dense layer (so negative flips to positive and thus supports the highest scoring class)?

Again, beware of such symmetry/invariability arguments when such symmetries/invariabilities are broken; and they are indeed broken here for a very simple reason (albeit hidden in the context), i.e. the specific one-hot encoding of the labels: we have encoded "cat" and "dog" as (say) [0, 1] and [1, 0] respectively, so, since we are interested in these 1s (which indicate class presence), it makes sense to look for the positive activations of the (late) convolutional layers. This breaks the positive/negative symmetry. Should we had chosen to encode them as [0, -1] and [-1, 0] respectively ("minus-one-hot encoding"), then yes, your argument would hold, and we would be interested in the negative activations. But since we take the one-hot encoding as given, the problem is no longer symmetric/invariant around zero - by using the specific label encoding, we have actually chosen a side (and thus broken the symmetry)...


Why isn't the reverse KL divergence commonly used in supervised learning? Forward KL Divergence (also known as cross entropy loss) is a standard loss function in supervised learning problems. I understand why it is so: matching a known a trained distribution to a known distribution fits $P \log(P/Q)$ where $P$ is the known distribution.

Why isn't the reverse KL divergence commonly used in supervised learning?

It’s not an exhaustive answer to your question, but here some aspects that might be helpful:

A common problem in supervised learning where you will see KL-divergence used are classification tasks. Very often in those cases the data points in the training set are assumed to belong to a single class $c_i$ where $i$ is from some index set $I$. The class membership is represented by one-hot encoding, which corresponds to a distribution $P$ with $p_i = 1$ for the class the data point belongs to and $p_j = 0$ for all $i \neq j$. The loss function mostly used in this problems is the average categorical cross-entropy $\langle E_P[-log(Q)]\rangle_{\rm data} = \langle H(P) - D(P || Q)\rangle_{\rm data}$, which in the case of single-class membership reduces to the binary cross-entropy. Because only one probability in $P$ is non-zero in this case, the cross-entropy simplifies to $\langle -\log(q_i)\rangle_{\rm data}$. The reverse KL-Divergence $D(Q || P)$ is strictly speaking not even defined in this case, as $Q$ needs to be absolute continuous w.r.t. $P$.

There are of course cases were class membership is probabilistic also in the training data and the above argument doesn't apply in full. From a more conceptual point of view, this thread summaries nicely the relation of cross-entropy to maximizing the likelihood of the observed data under the model. Using the reverse cross-entropy would maximize the likelihood of a typical sample drawn from the model under the (empirical) distribution of the observed data.

A small side note when you read the above link is that in the above described classification problem we have a probability model for the class membership for each point in the input space, this is why the average cross-entropy is minimized.

Classification problems are just one example of supervised learning, so this answer might not fully cover your question.


Making generated texts from "data-to-text" more variable I am diving in data-to-text generation for long articles (> 1000 words). After creating a template and fill it with data I am currently going down on paragraph level and adding different paragraphs, which are randomly selected and put together. I also added on a word level different outputs for date, time and number formats.
The challenge I see is, that when creating large amounts of such generated texts they become boring to read as the uniqueness for the reader goes down.
Furthermore, I also think it's easy to detect that such texts have been autogenerated. However, I still have to validate this hypotheses.
I was wondering if there is an even better method to bring in variability in such a text?
Can you suggest any methods, papers, resources or share your experience within this field.
I highly appreciate your replies!

The state of the art in text generation is the GPT model. GPT-3, which was just released in summer of 2020, has been used to generate many very impressive articles, and is widely considered the best text generation model. This article and this one should give you an example of how powerful it is at text generation.
GPT is a transformer based architecture, somewhat like BERT. The main difference is it only takes into account left context, which is why it is so well suited for text generation.
GPT-3 is still very new and is not available for free. However, GPT-2, the previous release of the model, is available for free. While obviously not as advanced as GPT-3, it is still quite impressive in its own right, and for someone trying to generate text, it is the clear choice.
Here is a link to a tutorial explaining the basics to get you started with GPT-2.
If you are interested in diving into the relevant research papers:
Here are the open AI papers on GPT 1-3

Improving Language Understanding by Generative Pre-Training
Language Models are Unsupervised Multitask Learners
Language Models are Few-Shot Learners

Additionally, if you have never seen transformers before, take a look at:

Attention Is All You Need


Which Rosenblatt's paper describes Rosenblatt's perceptron training algorithm? I struggle to find Rosenblatt's perceptron training algorithm in any of his publications from 1957 - 1961, namely:

Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms
The perceptron: A probabilistic model for information storage and organization in the brain
The Perceptron — A Perceiving and Recognizing Automaton

Does anyone know where to find the original learning formula?

The paper (or report) that formally introduced the perceptron is The Perceptron — A Perceiving and Recognizing Automaton (1957) by Frank Rosenblatt. If you read the first page of this paper, you can immediately understand that's the case. In particular, at some point (page 2, which corresponds to page 5 of the pdf), he writes

  Recent theoretical studies by this writer indicate that it should be feasible to construct an electronic or electromechanical system which will learn to recognize similarities or identities between patterns of optical, electrical, or tonal information, in a manner which may be closely analogous to the perceptual processes of a biological brain. The proposed system depends on probabilistic rather than deterministic principles for its operation, and gains its reliability from the properties of statistical measurements obtained from large populations of elements. A system which operates according to these principles will be called a perceptron.

See also Appendix I (page 19, which corresponds to page 22 of the pdf).

The paper The perceptron: A probabilistic model for information storage and organization in the brain (1958) by F. Rosenblatt is apparently an updated and nicer version of the original report.

A more accessible (although not the most intuitive) description of the perceptron model and its learning algorithms can be found in the famous book Perceptrons: An Introduction to Computational Geometry (expanded edition, third printing, 1988) by Minsky and Papert (from page 161 onwards).


MCTS: How to choose the final action from the root When the time allotted to Monte Carlo tree search runs out, what action should be chosen from the root?

The original UCT paper (2006) says bestAction in their algorithm.

Monte-Carlo Tree Search: A New Framework for Game AI (2008) says

The game action finally executed by the program in the actual game, is the one corresponding to the child which was explored the most.

A Survey of Monte Carlo Tree Search Methods (2012) says

The basic algorithm involves iteratively building a search tree until some predefined computational budget – typically a time, memory or iteration constraint – is reached, at which point the search is halted and the best-performing root action returned.
[...] The result of the overall search a(BESTCHILD(v0)) is the action a that leads to the best child of the root node v0, where the exact definition of “best” is defined by the implementation.
[...] As soon as the search is interrupted or the computation budget is reached, the search terminates and an action a of the root nodet0is selected by some mechanism. Schadd [188] describes four criteria for selecting the winning action, based on the work of Chaslot et al [60]:

Max child: Select the root child with the highest reward.

Robust child: Select the most visited root child.

Max-Robust child: Select the root child with both the highest visit count and the highest reward. If none exist, then continue searching until an acceptable visit count is achieved [70].

Secure child: Select the child which maximises a lower confidence bound.

[...] Once some computational budget has been reached, the algorithm terminates and returns the best move found,corresponding to the child of the root with the highest visit count.
The return value of the overall search in this case is a(BESTCHILD(v0,0)) which will give the action a that leads to the child with the highest reward, since the exploration parameter c is set to 0 for this final call on the root node v0. The algorithm could instead return the action that leads to the most visited child; these two options will usually – but not always! – describe the same action. This potential discrepancy is addressed in the Go program ERICA by continuing the search if the most visited root action is not also the one with the highest reward. This improved ERICA’s winning rate against GNU GO from 47% to 55% [107].

But their algorithm 2 uses the same criterion as the internal-node selection policy:
$$\operatorname{argmax}_{v'} \frac{Q(v')}{N(v')} + c \sqrt{\frac{2 \ln N(v)}{N(v')}}$$
which is neither the max child nor the robust child! This situation is quite confusing, and I'm wondering which approach is nowadays considered most successful/appropriate.

By far the most commonly used strategy is to select the child with the highest number of visits. This is as described in the 2008 paper you linked. It's also what's referred to as the "robust child" in the 2012 paper you linked.

In algorithm 2 of the 2012 paper, they actually use the highest average reward, which corresponds to "Max child". It looks like they're using the UCB1 policy, but they actually use a value of $0$ for the exploration parameter $c$, which makes the entire square root term drop out. This is also explained in the text at the end of your quote. But usually, a robust child / max visit count performs better.

Progressive Strategies for Monte-Carlo Tree Search is a different paper from 2008, in which these strategies are experimented with a bit. Usually, they perform similarly, but a robust child tends to be the best if there is any difference at all.


How can thousand-robot swarm coordinate their moves without bumping into each other? How can a swarm of small robots (like Kilobots) walking close to each other achieve collaboration without bumping into each other? For example, one study shows programmable self-assembly in a thousand-robot swarm (see article & video) which are moving without GPS-like system and by measuring distances to neighbours. This was achieved, because the robots were very slow.

Is there any way that similar robots can achieve much more efficient and quicker assembly by using more complex techniques of coordination? Not by walking around clock-wise (which I guess was the easiest way), but I mean using some more sophisticated way. Because waiting half a day (~11h) to create a simple star shape using a thousand-robot swarm is way too long!

There has been quite a few approaches to achieve such kind of distributed coordination. I present here one of them, for its generality and simplicity (that makes it easy to remember too). But first, the general idea behind these approaches is pretty interesting, around a mechanism called stigmergy.

Stigmergy is a behaviour coordination mechanism mediated by the environment. It was first described for termites, and the most famous example pertains to ants. Ants form trails when going out for food, but they often do not interact directly. It turns out they leave pheromones on the ground as they walk away from their hills. The pheromone allows them to find their way home, and it also guides where they are going: If they find a pheromone trace from one of their peers, they follow it and their own pheromones add up, reinforcing the signal of the trail. In stage, more and more ants get "together as they move", forming a trail. Coordination has been achieved.

Among the various implementation derived from stigmergy, there is the "field-based motion coordination model" (FBMCM). The idea is to create a (maybe virtual) environment that maintains some states of the world. Each object registers in the environment a signal that is maximum at the object position (its edges) and then decreases with distance. Moving objects (e.g. robots) each emit a signal relayed by the environment. They can then sense each other's field and act accordingly: E.g., when signals are strong, move away; when weak, it is safe to get closer, etc. Several complex group moves have been demonstrated in software simulators (platoon formation in games, drill simulations) and with robots. The benefit of this approach is that it can be cheap to compute even complex behaviours. For example, avoiding clashes requires simple "logic" code based on summing-up nearby fields value. FBMCM is pretty slick, used in video-games, but hard to implement in physical settings (to my dating knowledge), as it can be challenging to build a reliable environment. See for example the work from Mamei and Zambonelli, as well as one of the first industrial implementations for robots by Weyns et al.. Note that the implementation for robots required significant work on the environment infrastructure, made somewhat more feasible as it was a controlled warehouse.

The advantage of stigmergy-like models is that they are often simple and resilient: You can lose an ant without impact on the food-finding trail. On the downside, these models are usually slow, as the coordination takes time to emerge from indirect interactions. This can be improved upon by adding extra direct interactions (e.g. empowering ants with a GPS and a grocery store map, or just a magnetic-North sense).

In practice, these models can collapse if the environment implementation is not reliable. It can be difficult for robots or, say, self-driving cars, if they expect some transponders put on their way, as these devices are expensive to set and maintain, and they can be broken or stolen. It would be better to endow robots with radars, sonars or other proximity sensors to implement stigmergic models. One related example is the decision by Tesla to add radars to its cars, instead of assuming reliable transponders on the road (note: This is just a parallel; there is no official relation).

Other implementations and related models are, for example, tuple-based coordination languages such as Linda, and network protocols like Chord. As you see, these works are not necessarily in the "AI domain".


How can I detect diagram region and extract (crop) it from a research paper How can I detect diagram region and extract(crop) it from a research paper

I will consider that you need to extract(crop) the digram from the pdf research paper. You can use PyPDF2 or PyMuPDF to extract the images from the PDF file and then you can apply machine learning to do recognition and classification of the images. There are different types of machine learning solutions for image classification and you can start with Convolutional Neural Network and you can start here.

For more information for Recognition and classification of figures  see these  and to extract of figures from scholarly documents read this


Algorithm For Making Balanced Weapons In A Game? I am trying to make balanced weapon pairs. So there are five stats per weapon, and I am simulating a number of combats (1000) with different stats randomized, and counting the win, lose, and draw of the "weapon fight" for the database. I want an algorithm for making weapon1win-weapon2win as small as possible for balance through changing the weapon stats.

What happens:

Random Stats --> Combat 1000 Times --> Count Win & Lose --> Data For Training The AI

Data Sample:

{[(1,2,3,4,5) --> Weapon1Stats,(5,4,3,2,1) -->Weapon2Stats,1 --> WinWeapon1-WinWeapon2],[....]} (The text isn't part of the data sample, they are just there to help you know which variable is which. Also, the {[( s are all supposed to be [ s but changed for clarity)

I would like an function, preferably in C++ or Python, but just with text that is well explained is fine, for handling the data. The result would be a method to determine how to minimize WinWeapon1-WInWeapon2.

(edit)
I would say that the one I was looking for is one with something like a score for the weapon as in strength (1*stat1 + 2*stat2 etc...) but I do want something new that works better, I am also having problems creating leeway for the functions coefficients.(edit end)

I'm going to start by trying to restate your problem as I understand it.

You have a game which contains weapons.
Weapons are characterized by 5 different numbers, which can range over different values (1-5 in your examples?).
You have a way to simulate combat involving the two weapons.
The combat is random, but can be repeated many times. An average win rate can be determined.
You are looking for an AI algorithm that would take in a lot of pairs of statistics, along with the average win rates for one over the other, and give you insight into how to make the average win rate as close to 50% as possible.

If this sounds right, then fundamentally your problem is a form of regression, which something you could use AI for, but probably don't need to. However, your problem is probably not linear, so you need the interactions between the features. Here's what I suggest:

For each pair of weapons, store a comma separated list consisting of the stats for each weapon (one by one), followed by wins1 - wins2. At the top, list out the names of each attribute, separated by commas, (e.g. weapon1Str, weapon1Range, ... ,weapon1-weapon2 Then use a language like R that has simple support for complex forms of regression.

In R, this is then as simple as:

data <- read.csv(file="Myfile.csv")
lm(formula = dist ~ .*., data = data)

This should produce a list of "coefficients", one for each of the attributes, and one for the interaction between each pair of attributes, which form a lengthy quadratic equation in 10 variables.

Any zero of that equation should be a pair of weapons that minimizes this difference.

That's probably the place to start. If it doesn't work out, maybe come post a different question and we can help more.


Histopathological image vs. natural image What is the difference between a histopathological image and a natural image when training a neural network?

Histopathological images compared to natural images difference so, that in histopathological  images the image needed to be modeled may contain millions of pixels, whereas a recognition from natural image that this is a dog / house / certain person needs significantly less information to be extracted from the picture.

Histopathological images use WSIs (Whole slide image) which contain the part of tissue as a whole. It needs to be split to 256 x 256 pixel patches and those are used for ROI hunting (Region of Interest) and other analysis one by one.

Histopathological images (WSI) are quite rare nowadays and hold privately. More open data would rise the accuracy of machine learning and increase the amount of results that could be found from the data.

Complete source:

https://arxiv.org/pdf/1709.00786v1.pdf


Why are the Bellman operators contractions? In these slides, it is written
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{9} \label{9} \\
\|T V-T U\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \tag{10} \label{10}
\end{align}
where

$F$ is the space of functions on domain $\mathbb{S}$.
$T^{\pi}: \mathbb{F} \mapsto \mathbb{F}$ is the Bellman
policy operator
$T: \mathbb{F} \mapsto \mathbb{F}$ is the Bellman
optimality operator

In slide 19, they say that equality $9$ follows from
\begin{align}
{\scriptsize
\left\|
T^{\pi} V-T^{\pi} U
\right\|_{\infty}
=
\max_{s} \gamma \sum_{s^{\prime}} \operatorname{Pr}
\left(
s^{\prime} \mid s, \pi(s)
\right)
\left|
V\left(s^{\prime}\right) - U
\left(s^{\prime}\right)
\right| \\
\leq \gamma \left(\sum \operatorname{Pr} \left(s^{\prime} \mid s, \pi(s)\right)\right) \max _{s^{\prime}}\left|V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right| \\
\leq \gamma\|U-V\|_{\infty}
}
\end{align}
Why is that? Can someone explain to me this derivation?
They also write that inequality \ref{10} follows from
\begin{align}
{\scriptsize
\|T V-T U\|_{\infty}
= \max_{s}
\left|
\max_{a}
\left\{
R(s, a) + \gamma \sum_{s^{\prime}} \operatorname{Pr}
\left(
s^{\prime} \mid s, a
\right) V
\left(
s^{\prime}
\right)
\right\}
-\max_{a} \left\{R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) U\left(s^{\prime}\right)\right\} \right| \\
\leq \max _{s, a}\left|R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right)
-R(s, a)-\gamma \sum \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right) \right| \\
=
\gamma \max _{s, a}\left|\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right)\left(V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right)\right| \\
\leq \gamma\left(\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right)\right) \max _{s^{\prime}}\left|\left(V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right)\right| \\
\leq
\gamma\|V-U\|_{\infty}
}
\end{align}
Can someone explain to me also this derivation?

The inequality
\begin{align}
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty} & \leq \gamma\|V-U\|_{\infty} \label{1}\tag{1},
\end{align}
where $U$ and $V$ are two value functions, follows from the definition of Bellman policy operator (at slide  16)
\begin{align}
T^{\pi} V(s)
&\triangleq
R(s, a)+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, a\right) V\left(s^{\prime}\right) \\
&=R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) V\left(s^{\prime}\right), \; \forall s \in S
\tag{2}\label{2},
\end{align}
where $\triangleq$ means "defined as". Note the $\pi$ in the definition, hence the name Bellman policy operator (BPO), and note that the BPO holds for all $s$.
To prove (\ref{1}), first recall that
$$\left\|\mathbf {x} \right\|_{\infty } \triangleq \max _{i}\left|x_{i}\right|.$$
In the case of value functions $V$ and $U$, we have
$$\left\|V - U \right\|_{\infty } \triangleq \max_{s \in S}\left|V(s) - U(s) \right|.$$
Note also that $Pr$ is always non-negative (specifically, between $0$ and $1$).
Successively, we expand the left-hand side of (\ref{1}) by applying the definition (\ref{2}) and using the properties just mentioned
\begin{align}
{\scriptsize
\left\|T^{\pi} V-T^{\pi} U\right\|_{\infty}
=
\left\|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) - \left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right\|_{\infty}
}%
=\\
{\scriptsize
\max_{s \in S}
\left|
\left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) \right) - \left(
R(s, \pi(s))+\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right) \right)
\right|
}%
=
\\
\max_{s \in S}
\left|
\gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \gamma \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
\gamma
\max_{s \in S}
\left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) V\left(s^{\prime}\right) - \sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s) \right) U\left(s^{\prime}\right)
\right|
=
\\
\gamma \max_{s \in S} \left|
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left ( V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right)
\right|
=
\\
\gamma \max_{s \in S}
\sum_{s^{\prime}} \operatorname{Pr}\left(s^{\prime} \mid s, \pi(s)\right) \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|
\end{align}
Now, note that

$\max _{s^{\prime}}\left|V\left(s^{\prime}\right)-U\left(s^{\prime}\right)\right| = \| V - U \|_{_{\infty}}$
$\alpha x + (1- \alpha) x = x$ for $\alpha \in [0, 1]$
$\left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right| \leq \max_{s' \in S} \left| V\left(s^{\prime}\right) - U\left(s^{\prime}\right) \right|$

NOTE: I am not completely sure about all these steps. I need to think about them a little bit more (but later). I encourage you to try to spot wrong derivations!!


Which is more memory efficient uninformed or informed search algorithms? I have extensively researched now for three days straight trying to find which algorithm is better in terms of which algorithm uses up more memory. I know uninformed algorithms like depth-first search and breadth-first search do not store or maintain a list of unsearched nodes like how informed search algorithms do. But the main problem with uninformed algorithms is they might keep going deeper, theoretically to infinity if an end state is not found but they exist ways to limit the search like depth-limited search.

So am I right in saying that uninformed search in better than informed search in terms of memory with respect to what I said above?

Can anyone provide me with any references that show why one algorithm is better than the other in terms of memory?

Uninformed Search Techniques

Breadth-First Search needs to store a frontier of nodes to visit next (where "visit" basically means: see if it's the goal, generate its children and add to frontier otherwise). You can visualize the memory requirements of this as a pyramid; initially you just have the root node in there. As the search process continues and you keep going further down, it will become wider and wider (more memory required). The amount of memory required in the worst case to find a node at depth d can be expressed as O(b^d), where b is the branching factor (see details here). Intuitively, whenever you go one level deeper, your memory requirements multiply by a factor of b (your pyramid becomes b times wider; actually this means pyramid is not quite the right shape, it should rapidly curve outwards and become wider much faster than a triangle/pyramid would).

Depth-First Search also stores a frontier (or stack in the case of DFS) of nodes to visit next. However, in the case of DFS this generally grows less quickly. Whenever you visit a node (pop it off of the stack), you generate all of its children and push them on top of the stack. However, whereas BFS would then continue "to the right" by visiting a relatively "old" node and generating all of its children again, DFS continues by going "down" and visiting one of the children that only just got pushed onto the stack. Once DFS has finished a part of the search tree (you can visualize this in your head as DFS having completed searched for example the left half of the pyramid), that entire section no longer needs to be stored in memory. Assuming you have a search tree of finite depth d, the worst case space complexity is O(bd) (you have to search one path all the way down to a level of d, and for each of those levels have b nodes in the stack).

Note that, based on the above, it is not sufficient to characterize your comparison as "uninformed" vs "informed" search. We've only just looked at uninformed search techniques, and already have two different memory requirements.

In the above, I also did not consider subtleties such as the question of whether or not you're additionally memorizing which nodes of a graph you have already previously visited so that you can avoid visiting them again. This is not necessary in a tree, but may be necessary in a graph with cycles.

Informed Search Techniques

A* is probably one of the most canonical examples of informed search algorithms. In terms of worst case memory requirements, it's really similar to BFS; it also stores a frontier of nodes to visit next, but prioritizes those based on some estimate of "goodness" rather than Breadth-First order. In cases where you the information you're using (typically a combination of known/incurred costs + heuristic estimate of future costs) is of extremely poor quality, this can regress to the level of Breadth-First Search, so the worst-case space complexity is the same as that for BFS (see details). In practice, if you have high quality information (good heuristics), the memory requirements will be much better though; in the case of an ideal/perfect heuristic, your search algorithm will pretty much go directly to the goal and hardly require any memory.

There is also an algorithm called Iterative Deepening A* which has a much better space complexity, at the cost of often being slower. It is still an informed search algorithm using exactly the same kind of information as A* though. So, really, memory requirements cannot be characterized in terms of informed vs. uninformed search; in both cases there are algorithms that require a lot of memory, and algorithms that require less memory.


confusion matrix is useful to be applied to the result of fuzzy c means clustering or not After obtaining the result of algorithm fuzzy c means on the iris flower data set. Results are to be checked for their correctness. So, how to derive it in matlab. I mean code for that. please suggest.

Yes, you can use confusion matrix as a performance metric for a classifier where the output of the classifier is class labels.

If your classifier output is a likelihood floating point value for each of the classes, then you have to use some scheme of converting the likelihoods to a class label. For example, use softmax function for one hot encoding.

So, for the Fuzzy K Means clustering based classifier, you'll have to pick output class label as the 'index' if the Fuzzy cluster with maximum likelihood/ membership score. This will constitute predicted labels.

Using array of test labels and predicted labels to compute confusion matrix is straightforward in any computational platform, including Matlab of course.

-
Fuzzy K Means (also called fuzzy c means or FCM) is an unsupervised learning method. This means the assignment of training data points to clusters is not based on their class labels. Therefore, FCM is not used as a discriminative classifier.

Typically, FCM is used as an intermediate module to encode data. The encoded representation is used as a feature vector, in supervised learning, to train a discriminative classifier, such as a SVM.

This is not to say you can not use FCM for classification, but it would only work when data points of the same label occur in reasonably distinct clusters. This almost never happens with real world data and so it's recommended not to use FCM for classification.


How does one characterize a neural network with threshold-based activation functions? In an attempt at designing a neural network more closely modeled by the human brain, I wrote code before doing the reading. The neuron I have modeled operates on the following method.

Parameters: potential, threshold, activation.
[activation] = 0.0
Receive inputs, added to [potential].
If ([potential] >= [threshold])

[activation] = [potential]
[potential] = 0.0

Else

[potential] *= 0.5

In short, the neuron receives inputs, and decides if it "fires" if the threshold is met. If not, the input sum, or potential, decreases. Inputs are applied by adding their values to the input potentials of the input neurons, and connections multiply neuron activation values by weights before applying them to their destination potentials. The only difference between this an a spiking network is the activation model.

I am, however, beginning to learn that Spiking Neural Networks (SNNs), the actual biologically-inspired model, operate quite differently. Forgive me if my understanding is terribly flawed. I seem to have the understanding that signals in these networks are sharp sinusoidal wave-forms with between 100 and 300 "spikes" in a subdivision of "time," given for 1 "second." These signals are sampled for the "1 second" by the neuron, and processed by a differential equation that determines the activation state of the neuron. Synapses seem to function in the same manner -> multiplying the signal by a weight, but increasing or decreasing the period of the graph.

However, I wish to know what form of neuron activation model I created. I have been unable to find papers that describe a method like this.

EDIT. The "learnable" parameters of this model are [threshold] of the neuron and [weight] of the connections/synapses.

The model you describe is a kind of a leaky integrate-and-fire (LIF) neuron (see p. 7). It is leaky because the membrane potential decreases steadily in the absence of input. In contrast, in the simple integrate-and-fire (IF) model the membrane potential is retained indefinitely until the neuron spikes, at which point it is reset to 0. However, LIF neurons are usually modelled with exponential decay of the membrane potential, where you have a time constant $\tau$ and you compute the potential $P_{t}$ at time $t$ based on the potential $P_{t_{last}}$ at time when the last input arrived as

$P_{t} = P_{t_{last}} exp(- \frac{t - t_{last}}{\tau})$

This is the same formula as radioactive decay (see here for more details). The idea is that this model is inherently 'aware' of time, whereas the IF model (and your design above) do not factor in the timing of the spikes, so they act like a classical neural network activation. In any case, whether or not a neuron would fire does depend on the firing threshold, so I think that treating the threshold as a learnable parameter is justified - you just have to decide what rules to use for updating it.

Based on what you describe as your understanding of spiking neural networks, it seems that you have been reading about the Hodgkin-Huxley (HH) model (also in that paper I linked to). (Please correct me if I'm wrong.) You are correct in thinking that spikes in the brain are not infinitely narrow like a delta function but more like a very sharp sinusoidal signal, and the HH model faithfully reproduces that. However, the reason why the HH model is not actually used for simulations is that it is computationally very taxing. In practice, in most cases we do not actually care about the state of the neuron between inputs, as long as your model accurately describes the neuron state and what happens to it when an input arrives.

There are other models that approximate the HH model very closely but are much faster to simulate (like the Izhikevich model). However, the LIF model is very fast and sufficient in most cases.

Hope this helps!


What should the output of a neural network that needs to classify in an unsupervised fashion XOR data be? XOR data, without labels:
[[0,0],[0,1],[1,0],[1,1]]

I'm using this network for auto-classifying XOR data:
H1  <-- Dense(units=2, activation=relu)    #any activation here
Z   <-- Dense(units=2, activation=softmax) #softmax for 2 classes of XOR result
Out <-- Dense(units=2, activation=sigmoid) #sigmoid to return 2 values in (0,1)

There's a logical problem in the network, that is, Z represents 2 classes,
however, the 2 classes can't be decoded back to 4 samples of XOR data.
How to fix the network above to auto-classify XOR data, in unsupervised manner?

How to fix the network above to auto-classify XOR data, in unsupervised manner?

This cannot be done, except accidentally.
Unsupervised learning cannot replace or emulate supervised learning.
As a thought experiment, consider why you would expect the network to discover XOR, when simply considering outputs rounded to binary, you could equally find AND, OR, NAND, NOR or any of the 16 possible mapping functions from input to output. All of the possible maps are equally valid functions, and there is no reason why a discovered function mapping should become any one of them by preference.
Unsupervised learning approaches typically find patterns that optimise some measure across the dataset without using labelled data. Clustering is a classic example, and auto-encoding is sometimes considered unsupervised because there is no separate label (although the term self-supervised is also used, because there is still technically a label used in training, it happens to equal the input).
You cannot use auto-encoding approaches here anyway, because XOR needs to map $\{0,1\} \times \{0,1\} \rightarrow \{0,1\}$
You could potentially use a loss function based on how close to a 0 or 1 any output is. That should cause the network to converge to one of the 16 possible binary functions, based on random initialisation. For example, you could use $y(1-y)$ as the loss.


What algorithms are used for segmentation and classification of non solid regions in an image? In the process of segmentation, pixels are assigned to regions based on features that distinguishes them from the rest of the image. Value Similarity and Spatial Proximity, for example, are two important principles that assume that points in the same region will have pixels that are spatially close and have similar values.

In lots of situations this is true, but what about regions composed of pixels that are not similar in value? Consider the image below. The same "logical" region is composed of different elements that together represent something meaningful. In the same region there are trees with different sizes and shapes, with shadow over some of them etc. There are different things, with pixels that differ a lot in value, but I still need to group them together in the same region. From the image you can see that I don't care so much with differences in color. In this case texture is the most important attribute. What algorithms are used to do the segmentation and classification in problems like that?

I'm already looking for some algorithms and techniques that focus on texture, but some opinions from the experts will help me a lot. I think I need some orientation. Thanks!

If you have enough data, you can train a segmentation net example for a specific group of data (i.e. trees). Single DNN with multiple output branches can easily segment & classify the data that you are searching for.


Can an AI be made to maintain a train of thought? This mostly refers to human-like or chatbot AI, but could maybe be used in other applications (math or something?).

Basically, it occurred to me, that when I'm thinking or speaking, there is a constant feedback loop, in which I am formulating which words to use next, which sentences to form, and which concepts to explore, based on my most recent statements and the flow of the dialogue or monologue. I'm not just responding to outside stimulus but also to myself. In other words, I am usually maintaining a train of thought.

Can AI be made capable of this? If so, has it been demonstrated? And to what extent? While typing this, I discovered the term "thought vectors", and I think it might be related.

If I read correctly, thought vectors have something to do with allowing AI to store or identify the relationships between different concepts; and if I had to guess, I'd say that if an AI lacks a strong understanding of the relationships between concepts, then it would be impossible for it to maintain a coherent train of thought. Would that be a correct assumption?

(ps. in my limited experience with AI chatbots, they seem to be either completely scripted, or otherwise random and often incoherent, which is what leads me to believe that they do not maintain a train of thought)

First, for almost every question of the form "Can AI be made to X", the most obvious and straightforward answer is something like "We don't know. Probably, but if it hasn't been done yet, we're really not sure."

It's also important to understand that, from a technology standpoint, there isn't one "thing" called "AI".  There are many, many different technologies, which are loosely related (at best) and are generally lumped together under the overall rubric of "Artificial Intelligence".

All of that said, yes, there has been work on adding memory, even long-term memory, to various kinds of "AI".  The most notable recent example is the advent of LSTM in recurrent neural networks.

Additionally, some of the work done on "cognitive architectures" has focused on the use of memory.  For more info on that, look up ACT-R and/or SOAR and read some of those papers.

What isn't clear to me offhand, is whether or not anybody has tried applying any of these techniques to chat-bots in particular.  I wouldn't be surprised if somebody had, but I can't cite any such research off the top of my head.


How do Artificial Neural Networks store data compared to Biological Neural Networks? Do scientists know by what mechanism biological brains/biological neural networks store data?

I was thinking about @kenorbs question about implanting nanobots to build an AGI on top of human wetware.

I only have a vague notion that we store data in our brains by altering synapses?

Links, Criticism and Detailed Explanation welcome.

I also would love a decent description of how a vanilla Artificial Neural Network stores data.

Questions:

How is data stored in a biological Neural Network?
How is data stored in an Artificial Neural Network?

Second question first: Data is stored in an ANN in the form of weights in the adjacency matrix between neurons. During training, these weights are updated by a learning algorithm (such as backpropagation).

First question: according to award-winning neuroscientist Tim Bliss:

  “It’s been accepted really since the turn of the 20th century, since the time of the Spanish neuroscientist Ramón y Cajal, that really the only place where memories can be stored is at synapses, the junctions between nerve cells.

A protein called the NMDA receptor plays a key roll in the strengthening of synaptic connections (which is more broadly achieved by a form of Hebbian Learning).


Is it necessarry to create theory/infrastructure to prevent people from creating AI incompatible with a safe model (if we create one)? Nowadays we don't know how to create AI in a safe way (I think that we don't even know yet how to define a safe AI), but there is a lot of research in developing a model allowing it.

Let's say, that someday we discover such a model (maybe even it would be possible to mathematically prove its safety). Is it rational to ask, how do we prevent people from creating AI outside of this model (e.g. they are so confident in their own model, that they just pursue it and end up with something like paperclip scenario)?

Should we also think about creating some theory/infrastructure preventing such a scenario?

It seems impossible to prevent that. If someone can make a safe AI from scratch in the near future, then someone else can make a dangerous AI from scratch as well. If all that's needed is a computer (or eventually a robot) it will be really hard to stop people from creating one.

Banning computers? Maybe it could prevent it, but that comes with quite a few negative sides as well.

A law against creating AIs? Would be really hard to follow-up, and what about the AIs people want to make to be used for something "good"?

And even if we could come up with a good set of laws they probably wouldn't be introduced in all countries, and also be really hard to follow up.

I guess we just need to make our own defence mechanisms which can fight unsafe AIs (physically or virtually) when that time comes.


Is there any opensource 2d open-world simulation with python API? For my pet project I’m looking for a grid-like world simulation with some kind of resources that requires from agent incrementally intelligent behaviour to survive.

Something like this steam game, but with API.  I’ve seen minecraft fork, but it’s too complex for my task. There is pycolab, i can build some world on this engine, but I’d prefer ready-to-use simulations.

Is there any option? I'll appreciate any suggestion.

You could try Mesa.

It has various examples that are commonly-used in agent-based modelling, like Epstein's model, a wolf/sheep predator/prey model, and many more.

There is also an introductory tutorial.


Are there neural networks that accept graphs or trees as inputs? As far I know, the RNN accepts a sequence as input and can produce as a sequence as output.

Are there neural networks that accept graphs or trees as inputs, so that to represent the relationships between the nodes of the graph or tree?

Yes, there are numerous, coming under the umbrella term Graph Neural Networks (GNN).

The most common input structures accepted by these techniques are the adjacency matrix of the graph (optionally accompanied by its node feature matrix and/or edge feature matrix, if the graph has such information).

A Comprehensive Survey on Graph Neural Networks, Wu et al (2019) divides GNN's into four subgroups:

Recurrent graph neural networks (RecGNN)
Convolutional graph neural networks (ConvGNN)
Graph autoencoders (GAE)
Spatial-temporal graph neural networks (STGNN)

ConvGNN's can themselves be classified by whether they use Spectral methods or Spatial methods, and GAE's by whether they are designed for Network embedding or Graph generation.


How are connection weights "perturbed"? I am reading through the NEAT paper here. On page 14 of the PDF, there is this quote about mutation:

  There was an 80% chance of a genome having its connection weights mutated, in which case each weight had a 90% chance of being uniformly perturbed and a 10% chance of being assigned a new random value.

What exactly does it mean to perturb weights? What is uniform vs. nonuniform perturbation?

Is there an established method to do this? I am imagining the process as multiplying each connection weight by a random number, but I'm unfamiliar with the term.

Perturbed here means adding a small random value to the weight. That random value comes from a uniform distribution or from a gaussian (or any distribution really).
Imagine just nudging the weight by a little.

It’s done to overcome the problem of local minima where models can get stuck with a good set of weights but not the best set of weights. By perturbing the weights a little, the model has a chance of finding a better set of weights. Searching for methods that deal with local minima like stochastic gradient descent will give you a better intuition.


Is it compulsary to normalize the dataset if doing so can negatively impact a Binary Logistic regression performance? I am using raw data set with 4 feature variables (Total Cholesterol, Systolic Blood Pressure, Diastolic Blood Pressure, and Cigraeette count) to do a Binominal Classification (find stroke likelihood) using Logistic Regression Algorithm.

I made sure that the class counts are balanced. i.e., an equal number of occurrences per class.

using Python + sklearn, the problem is that the classification performance gets very negatively-impacted when I try to normalize the dataset using

 X=preprocessing.StandardScaler().fit(X).transform(X)

or

  X=preprocessing.MinMaxScaler().fit_transform(X)

So before normalizing the dataset:

         precision    recall  f1-score   support

      1       0.70      0.72      0.71        29
      2       0.73      0.71      0.72        31

avg / total   0.72      0.72      0.72        60

while after normalizing the dataset (the precision of class:1 decreased significantly)
             precision    recall  f1-score   support

      1       0.55      0.97      0.70        29
      2       0.89      0.26      0.40        31

 avg / total  0.72      0.60      0.55        60

Another observation that I failed to find an explanation to is the probability of each predicted class.

Before the normalization:

 [ 0.17029846  0.82970154]
 [ 0.47796534  0.52203466]
 [ 0.45997593  0.54002407]
 [ 0.54532438  0.45467562]
 [ 0.45999462  0.54000538]

After the normalization ((for the same test set entries))

 [ 0.50033247  0.49966753]
 [ 0.50042371  0.49957629]
 [ 0.50845194  0.49154806]
 [ 0.50180353  0.49819647]
 [ 0.51570427  0.48429573]

Dataset description is shown below:

       TOTCHOL    SYSBP    DIABP  CIGPDAY   STROKE
count  200.000  200.000  200.000  200.000  200.000
mean   231.040  144.560   81.400    4.480    1.500
std     42.465   23.754   11.931    9.359    0.501
min    112.000  100.000   51.500    0.000    1.000
25%    204.750  126.750   73.750    0.000    1.000
50%    225.500  141.000   80.000    0.000    1.500
75%    256.250  161.000   90.000    4.000    2.000
max    378.000  225.000  113.000   60.000    2.000

SKEW is

TOTCHOL    0.369
SYSBP      0.610
DIABP      0.273
CIGPDAY    2.618
STROKE     0.000

Is there a logical explanation for the decreased precision?

Is there a logical explanation for the very-close-to-0.5 probabilities?

I found the answer to my question, went back to the Python script and in the command that fits the model i.e.

 LR = LogisticRegression (C=0.1, solver = "sag",max_iter=1000).fit (X_train, y_train)

The parameter C was set to 0.001 which is a very small value (meaning lambda is very high as C=1/lambda) (C is the regularization strength and smaller values indicate stronger regularization). more on that matter can be found here and here


In a face database containing multiple images per subject, how do we determine the face image which is most suited for face recognition? Let us imagine a face database with several subjects, each subject having multiple face images. How do we determine which is the best face suitable for face recognition purposes?

This should not matter that much as long as you do enough preprocessing?
But one could use e.g. the Facenet architecture to extract the embedding vector for each face of the same subject.
Then you could compute the covariance matrix and perform PCA on it. This would give you the most significant features with which you can decide which face to take.
Alternatively, you could do something in the direction of the eigenfaces: https://en.wikipedia.org/wiki/Eigenface.


What happens when you select actions using softmax instead of epsilon greedy in DQN? I understand the two major branches of RL are Q-Learning and Policy Gradient methods.
From my understanding (correct me if I'm wrong), policy gradient methods have an inherent exploration built-in as it selects actions using a probability distribution.
On the other hand, DQN explores using the $\epsilon$-greedy policy. Either selecting the best action or a random action.
What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?

DQN on the other hand, explores using epsilon greedy exploration. Either selecting the best action or a random action.

This is a very common choice, because it is simple to implement and quite robust. However, it is not a requirement of DQN. You can use other action choice mechanisms, provided all choices are covered with a non-zero probability of being selected.

What if we use a softmax function to select the next action in DQN? Does that provide better exploration and policy convergence?

It might in some circumstances. A key benefit is that it will tend to focus on action choices that are close to its current best guess at optimal. One problem is that if there is a large enough error in Q value estimates, it can get stuck as the exploration could heavily favour a current best value estimate. For instance, if one estimate is accurate and relatively high, but another estimate is much lower but in reality would be a good action choice, then the softmax probabilities to resample the bad estimate will be very low and it could take a very long time to fix.
A more major problem is that the Q values are not independent logits that define preferences (whilst they would be in a Policy Gradient approach). The Q values have an inherent meaning and scale based on summed rewards. Which means that differences between optimal and non-optimal Q value estimates could be at any scale, maybe just 0.1 difference in value, or maybe 100 or more. This makes plain softmax a poor choice - it might suggest a near random exploration policy in one problem, and a near determinitsic policy in another, irrespective of what exploration might be useful at the current stage of learning.
A fix for this is to use Gibbs/Boltzmann action selection, which modifies softmax by adding a scaling factor - often called temperature and noted as $T$ - to adjust the relative scale between action choices:
$$\pi(a|s) = \frac{e^{q(s,a)/T}}{\sum_{x \in \mathcal{A}} e^{q(s,x)/T}}$$
This can work nicely to focus later exploration towards refining differences between actions that are likely to be good whilst only rarely making obvious mistakes. However it comes at a cost - you have to decide starting $T$, the rate to decay $T$ and an end value of $T$. A rough idea of min/max action value that the agent is likely to estimate can help.


What are good action outputs for reinforcement learning agents acting in a trading environment? I am trying to build an agent that trades commodities in a exchange setting. What are good ways to map the action output to real world actions? If the last layer is a tanh activation function, outputs range between [-1,+1]. How do I map these values to real actions? Or should I change the output activation to linear and then directly apply the output as an action?

So let's say the output is tanh activated and it's -0.4, 5. I could map this to:
- -0.4 --> sell 40% of my holdings for 5$ per unit
- -0.4 --> sell 40% for 5$ in total

if it was linear, I could expect larger outputs (e.g -100, 5). Then the action would be mapped to:
- sell 100 units for 5$ each
- sell 100 units for 5$ total

Working Backward

Working backward from the trading interface available to you Note 1, you will need two things for each Exchange-Traded Fund (ETF) or other tradable commodity of another class.

An operation to perform
An associated monetary amount

The system can have an output structure Note 2 for each ETF like this (depending of course on the trading interface available to you to you and your banks primary monetary system).

Ternary operation indicator (buy, sell, hold)
Trade amount in USD

Why Not Just One Number per ETF?

A few corroborating reasons exist for why a single positive, zero, or negative trade amount is not likely the optimal architectural choice.

The difference between holding and trading an amount of 1.0 monetary unit is not equivalent to the difference between trading 1.0 and 2.0 monetary units.  Stated mathematically, the function of profitability to trade amount is not smooth and probably not even continuous.
When you implement the ternary output as two binary outputs {buy, sell}, training against the Boolean expression (buy AND sell) is likely to improve your initial performance and possibly your ongoing performance Note 3.

Limitations on Real Trades to Consider

Because you have the limit of the assets in the liquid account from which you can buy, you will need to train against breaking the bank or using a stage after the NN outputs imposing rules or a formula based on gain and loss probabilities.  This financial constraint muddies your question because there are several ways to ensure you do not break the bank (get an insufficient funds response from your trade operation).

Optimizing a Deeper Architecture

Let's first assume you use probabilistic calculus to produce a closed form (formula) for how to trade based on predictions from the NN architecture you design.  Then the NN outputs might be continuous values representing the distribution of outcomes for each ETF.  Projections will almost always be dependent on investment duration.

In such an architecture the NN output activation scheme would be a continuous function (not necessarily linear) producing something like this Note 4.

Mean expected delta value in one day
Std deviation in one day expectation
Mean expected delta four weeks
Std deviation in two week expectation
Mean expected delta two years
Std deviation in two years expectation

Any NN optimization of an investment portfolio that does not inherently deal with probability is nonsense.  Optimizing for maximum gain will introduce great risk.  Optimizing for minimum risk could result in losses.  The goal of optimization must be some representation of the balance between the desire to win and the fear of loosing, to put it in anthropological terms.

Mean and standard deviation are obvious starting choices, and the traditional categories of short, medium, and longer term investment is also reasonable to begin with in the temporal domain Note 5.

Pure NN

Now let's assume you replace the calculus with another NN scheme trained to maximize portfolio total assets Note 6.  Such a replacement NN scheme must also take as an input your available liquid asset amount along with the above probabilistic projections.  You must also train to ensure the aggregation of buys and sells do not break your bank, that your liquid asset account never drops below zero.

The trade amount activation should be a continuous function, but not necessarily linear and probably not tanh either because the asymptotes of that function would be counterproductive unless it is made proportional to your available liquid assets to train by aggregating your options in your architecture.  However, that's not optimal because you may find a better deal to use those assets a minute later.

The odd roots (third root or fifth root or both with coefficients), when used along with training to not break the bank and to maximize the rate of portfolio growth will produce a better environment for learning in the earlier layers because of the probabilistic and aggregation realities of liquid asset limitations.

NOTES

Note 1 — Preferably a secure RESTful API, however experimentation can employ a web browser the HTTPS transactions you can control using https://github.com/SeleniumHQ/selenium or https://github.com/watir/watir.

Note 2 — Input would be things like number of companies, exposure ratings provided by investment firm(s), short options, inception date, and a sequence of events, each event containing fixed point numbers and flags like these.

Price per share
Closing price flag
Expense ratio
Dividend per share

Note 3 — The binary output vector {buy, sell} value of {1, 0} causes a buy.  The value of {0, 1} causes a sell.  The value {0, 0} is ignored, and {1, 1} may trigger another training operation to correct this illegal output, which is likely to be indicative of the staleness of the last round of training.  If the NN is re-entrant (reinforced), the feedback vector could include this anomalous flag or weight it heavily in an aggregation of feedback sources.  In summary, the ternary scheme can be expected to augment the training speed and resulting accuracy.  More importantly, it opens additional options for continuous optimization.

Note 4 — Delta is an aggregation of price increase or loss, dividends, and holding and trading expenses because that is the proper metric related to profitability of the portfolio.

Note 5 — Four weeks and two years have been chosen for the middle and longer range projections so that the time ratios are 26.09 and 28 between the three durations.  Common choices are temporally skewed.  If 1 day, 1 week, and 1 year had been chosen, the ratios would have been 7.0 and 52.18.  If 1 day, 1 month, and 1 year had been chosen, they would have been 30.44 and 12.

Note 6 — Do not assume that even a well trained NN will ever outperform formulae properly derived from probabilistic calculus for the last stage in a trading profitability architecture.


Expected SARSA vs SARSA in "RL: An Introduction" Sutton and Barto state in the 2018-version of "Reinforcement Learning: An Introduction" in the context of Expected SARSA (p. 133) the following sentences:

  Expected SARSA is more complex computationally than Sarsa but, in return, it eliminates the variance due to the random selection of $A_{t+1}$. Given the same amount of experience we might expect it to perform slightly better than Sarsa, and indeed it generally does.

I have three questions concerning this statement:

Why is the action selection random with Sarsa? Isn't it on-policy and therefore $\epsilon$-greedy?
Because Expected-Sarsa is off-policy the experience it learns from can be from any policy that at least explores everything in the limit e.g. random action-selection with equal probabilities for every action. How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an $\epsilon$-greedy policy, especially with the same amount of experience?
Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?

Why is the action selection random with Sarsa?

A policy could be stochastic. In the case of SARSA, it is stochastic because of the use of $\epsilon$-greedy.

  Isn't it on-policy and therefore ϵ-greedy?

I don't quite understand the question. SARSA is on-policy evaluation with $\epsilon$-greedy policy. Q-learning is off-policy evaluation with $\epsilon$-greedy policy. $\epsilon$-greedy is just a way to turn an action-value function into a policy.

  Because Expected-Sarsa is off-policy the experience it learns from can be from any policy ... How can Exected-Sarsa learning from such policy be generally better than normal Sarsa learning from an ϵ-greedy policy, especially with the same amount of experience?

It is unfair to compare different natures of experiences because off-policy experience contains less useful information. Thus, both SARSA and Expected SARSA should use their own on-policy experience for comparison.

While Expected SARSA update step guarantees to reduce the expected TD error, SARSA could only achieve that in expectation (taking many updates with sufficiently small learning rate). Through this perspective, there is little doubt that Expected SARSA should be better.

  Probably more general: How can on-policy and off-policy algorithms be compared in such way (e.g. through variance) even though their concepts and assumptions are so different?

Same as the previous answer, it is unfair to compare them without the same quality of experiences.


Calculating gradient for log policy when variance is not constant I've noticed that when modelling a continuous action space, the default thing to do is to estimate a mean and a variance where each is parameterized by a neural network or some other model.

I also often see that it is one network $\theta$ models both. The REINFORCE objective can be written as

$$\nabla \mathcal{J}(\theta) = \mathbb{E}_{\pi} [\nabla_\theta \log \pi(a_t|s_t) * R_t] $$

For discrete action space this makes sense since the output of the network is determined by a softmax. However, if we explicitly model the output of the network as a Gaussian, then the gradient of the log likelihood is of a different form,

$$\pi_\theta(a_t|s_t) = Normal(\mu_\theta(s_t), \Sigma_\theta(s_t))$$

and the log is:

$$\log \pi_\theta(a_t | s_t) =  -\frac{1}{2} (a_t-\mu_\theta)^\top \Sigma^{-1}_\theta(a_t-\mu_\theta) + \log 2 \pi \det({\Sigma_\theta})$$

In the slides provided here (slide 18):
http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching_files/pg.pdf

IF the variance is held constant, then we can solve this analytically:

$$\nabla_\theta \log \pi_\theta(a_t|s_t) = (a_t - \mu_\theta) \Sigma^{-1} \phi(s)$$

But, are things always modelled assuming a constant variance? If it's not constant then we have to account for the inverse of the covariance matrix
as well as the determinant?

I've taken a look at code online and from what I've seen, most of them assume the variance is constant.

@NielSlater

Using the reparameterization trick we would use a normal distribution with fixed parameters 0 and 1.
$$ a_t \sim \mu_\theta(s_t) + \Sigma_\theta(s_t) * Normal(0, 1) $$

Which is the same as if we had actually sampled directly from a distribution, $ \pi_\theta(a_t | s_t) = Normal(\mu_\theta(s_t), \Sigma_\theta(s_t))$ and let's us calculate the corresponding $\log \pi_\theta(a_t|s_t)$ and $\nabla_\theta \log \pi_\theta(a_t | s_t)$ without having to differentiate through the actual density.

It's true that computing the log prob of a sample from a gaussian requires inverting a matrix and dealing with the determinant in the general case of a full covariance matrix. If you wanted to backprop through the log prob, you'd need to backprop through these operations, which by the way are available as differentiable operations in pytorch. See the gaussian distribution class which deals with these operations: https://github.com/pytorch/pytorch/blob/master/torch/distributions/multivariate_normal.py#L177
You can see there's a half_log_det and although you won't find an explicit matrix inversion, you'll see that it uses a differentiable triangular linear solve (torch.trtrs) at some point, which is essentially doing a matrix inversion in a more efficient way.


Can Neural Networks self-optimize? Suppose that you show a neural network its own code, and allow it to edit itself?

Can a neural network modify its own weights and architecture (the number of layers, the number of neurons per layer, etc.)?

Without using back propagation and gradient descent.

Neural networks modify its own weights and architectures.

Is this a valid notion?  Is it being explored? Is this a valid concept in relation to NNs?

It feels like the ultimate thing to recognize and edit its own structure as data.

Since a neural network does iteratively learn its own weights I assume you mean the structure of the neural network - the number of layers and nodes per layer.

If what I said above was your question, then yes, it most definitely is being explored.
Even when a neural network is allowed to learn its own structure it still needs to be suited to a specific problem.
Google wrote a blog where they used machine learning to explore the architecture of a neural network calling it the AutoML project where the blog claimed "To make this process of designing machine learning models much more accessible, we’ve been exploring ways to automate the design of machine learning models."
It was found that AutoML can design small neural networks that perform on par with neural networks designed by human experts, but these results were constrained to small academic datasets like CIFAR-10, and Penn Treebank.

This study was further extended to more challenging problems and datasets in another blog where they proposed NASNet

Both these blogs were written in 2017 and I'm not very sure how much progress has been made on this since, but I hope these blogs helped!


How are the weights retained for filters for a particular class in a CNN? I am new to CNN. What I have learned so far about the filters is that when we are giving a training example to our model, our model updates the weights by gradient descent to minimize the loss function.
So my question is how the weights are retained for a particular class label?

The question is vague as my knowledge is vague. It's my 4th hour to CNN.

For example, if I am talking about the MNIST dataset with 10 labels. Let's say I am giving 1 image to my model initially. It will have a bigger loss for the forward pass. Let's say now it came for the back pass and adjusted the weights for and minimized the loss function for that label.
Now, when a new label arrives for training, how will it update the weights for filters which have already been updated according to the previous label?

Besides the last layer rest of the weights are shared among all classes. When an image is passed to the network all weights are updated accordingly. The only weights that are directly responsible for one specific class are the ones of the final layer. The rest of the weights are updated to find the best values to minimize the average loss for all classes.

To rephrase there aren't "filters" in convolutional layers that are specific for a single class. They are used to extract features so that the final layer can make the final prediction (which has weights for each specific class).

I'd suggest you look a bit into how gradient descent and backpropagation works.


Is it possible for value-based methods to learn stochastic policies? Is it possible for value-based methods to learn stochastic policies? I'm trying to get a clear picture of the different categories for RL algorithms, and while doing so I started to think about settings where the optimal policy is stochastic (POMDP), and if it is possible to learn this policy for the "traditional" value-based methods
If it is possible, what are the most common methods for doing this?

Is it possible for value-based methods to learn stochastic policies?

Yes, but only in a limited sense, due to the ways it is possible to generate stochastic policies from a value function. For instance, the simplest exploratory policy used by SARSA and Monte Carlo Control, $\epsilon$-greedy, is stochastic.

SARSA natually learns the optimal $\epsilon$-greedy policy for any fixed value of $\epsilon$. That is not quite the same as learning the optimal policy, but might still be useful in a non-stationary environment where exploration is always required and the algorithm is forever learning online.

You can also use other functions to generate stochastic policies from value functions. For instance, sampling from the Boltzmann distribution over action values using a temperature parameter to decide relative priorities between actions with different action values.

However, all these approaches share the problem that they cannot converge towards an optimal stochastic policy. The policies are useful for mangaging exploration, but will only be optimal in the limited sense of optimal given the fixed policy generator or by chance. There is no way for a purely value-based method to learn a conversion from values to an optimal balance of probabilities for action choice.

For strict MDPs this is not an issue. If the MDP has the Markov property in the state representation, then there will always be a deterministic optimal policy, and value-based methods can converge towards it. That may include reducing $\epsilon$ in $\epsilon$-greedy approaches or the temperature in Gibbs sampling, when using an on-policy method.

  I started to think about settings where the optimal policy is stochastic(POMDP), and if it is possible to learn this policy for the "traditional" value-based methods

It isn't.

To resolve this you need to add some kind of policy function and a mechanism to search for better policies directly by modifying that function. Policy Gradient methods are one approach, but you could include genetic algorithms or other search methods too under this idea.

It may still be useful to use a value-based method as part of a policy search, to help evaluate changes to the policy. This is how Actor-Critic works.


How can we compute the gradient of max pooling with overlapping regions? Studying CNN Back-propagation I can't understand how can we compute the gradient of max pooling with overlapping regions ?

That's also a question from this quiz and can be also found on this book .

When gradients in a neural network can follow multiple paths to same parameter, the different gradient values from the sources can often be added together, because the operations in the forward direction are also sums and $\frac{d}{dx}(y+z) = \frac{dy}{dx} + \frac{dz}{dx}$.

That is the case already with gradients of kernels (which are sums over the image area), and is equally the case for overlapping aggregation, including maximums, minimums or averages.

So in the 1d case, if you have a max pool over the input params $[a_0, a_1, a_2, a_3, a_4]$ a max function $m_0 = max(a_0, a_1, a_2)$, $m_1 = max(a_2, a_3, a_4)$ which overlap at $a_2$, and gradients $\nabla_{\mathbf{m}} J = [\frac{\partial J}{\partial m_0}, \frac{\partial J}{\partial m_1}]$, then you would allocate those gradients to vector $\mathbf{a}$ according to which items in each group was the max of that group, adding them when they overlapped.

Examples:

If $\mathbf{a} = [3,0,1,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0.7, 0, 0, 0.9, 0]$

If $\mathbf{a} = [3,0,4,2,0]$ and  $\nabla_{\mathbf{m}}J = [0.7, 0.9]$, then $\nabla_{\mathbf{a}}J = [0, 0, 1.6, 0, 0]$


What tools are used to deal with adversarial examples problem? The problem of adversarial examples is known to be critical for neural networks. For example, an image classifier can be manipulated by additively superimposing a different low amplitude image to each of many training examples that looks like noise but is designed to produce specific misclassifications.

Since neural networks are applied to some safety-critical problems (e.g. self-driving cars), I have the following question

What tools are used to ensure safety-critical applications are resistant to the injection of adversarial examples at training time?

Laboratory research aimed at developing defensive security for neural networks exists. These are a few examples.

adversarial training (see e.g. A. Kurakin et al., ICLR 2017)

defensive distillation (see e.g. N. Papernot et al., SSP 2016)

MMSTV defence (Maudry et al., ICLR 2018).

However, do industrial-strength, production-ready defensive strategies and approaches exist?  Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?
There are already (at least) two questions related to the problem of hacking and fooling of neural networks. The primary interest of this question, however, is whether any tools exist that can defend against some adversarial example attacks.

However, do industrial strength, production ready defensive strategies and approaches exist? Are there known examples of applied adversarial-resistant networks for one or more specific types (e.g. for small perturbation limits)?

I think it's difficult to tell whether or not there are any industrial strength defenses out there (which I assume would mean that they'd be reliable against all or most known methods of attacking). Adversarial Machine Learning is indeed a highly active, and growing, area of research. Not only are new approaches for defending being published quite regularly, but there is also active research into different approaches for "attacking". With new attack methods being discovered frequently, it's unlikely that anyone can already claim to have approaches that would work reliably against them all.

  The primary interest of this question, however, is whether any tools exist that can defend against some adversarial example attacks.

The closest thing to a ready-to-use "tool" that I've been able to find is IBM's Adversarial Robustness Toolbox, which appears to have various attack and defense methods implemented. It appears to be in active development, which is natural considering the area of research itself is also highly active. I've never tried using it, so I can't vouch personally for the extent to which it's easily usable as a tool for industry, or if it's maybe really only still suitable for research.

Based on comments by Ilya, other frameworks that may be useful to consider are Cleverhans and Foolbox.


Label arrangement for custom Keras image generator I am trying to generate 90 and 270 degrees rotated versions of my sample images on the fly during training. I found an example and modifying it. But I am confused about what should be the order? For instance in one batch I have 32 images and my image generator should return total 64 images. Let's say upper case letters are 90 degree and lower case letters are 270 degree rotated images. Should the order be AaBbCc or ABCabc? I apply the same to the validation set. Here is the related code fragment:

Edit: Code fragment added.

    def _get_batches_of_transformed_samples(self, index_array):
    # create list to hold the images
    batch_x = []
    # create list to hold the labels
    batch_y = []
    # rotation angles
    target_angles = [0, 90, 180, 270]
    angle_categories = list(range(0, len(target_angles)))
    self.classes = target_angles
    self.class_indices = angle_categories
    # generate rotated images and corresponding labels
    for i, j in enumerate(index_array):
        is_color = int(self.color_mode == 'rgb')
        image = cv2.imread(self.filenames[j], is_color)
        if is_color:
            image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

        for rotation_angle, cat_angle in zip(target_angles, angle_categories):
            rotated_im = rotate(image, rotation_angle, self.target_size[:2])
            if self.preprocess_func: rotated_im = self.preprocess_func(rotated_im)
            # add dimension to account for the channels if the image is greyscale
            if rotated_im.ndim == 2: rotated_im = np.expand_dims(rotated_im, axis=2)
            batch_x.append(rotated_im)
            batch_y.append(cat_angle)

    # convert lists to numpy arrays
    batch_x=np.asarray(batch_x)
    batch_y=np.asarray(batch_y)

    batch_y = to_categorical(batch_y, len(target_angles))
    return batch_x, batch_y

I actually rotate them as 0, 90, 180 and 270 degrees. As seen in the code, for each batch I return all the rotated versions of all the images in the batch. But is this correct or should I return first 0 degree rotated versions, second 90 degree rotated versions so on?

Edit2: I checked my previous work which I use the built in Keras ImageDataGenerator. generator.classes returns [zeros(100,1); ones(100,1)]. In that study I only have two classes. I understand that Keras indexes the images as [class1, class2, ...]. I think I have to do the same.

I am not allowed to comment, so I am adding it here. Not sure I understand, are you asking if the training data must be ordered?

If so, training data must be random, and not AaBbCc or ABCabc.
Here the input to the NN will be the source image, and a category 90 degrees or 270 degrees as too separate inputs. i.e. inputs [0...k][k+1][k+2], where 0 - k are normalised pixel data, k+1 is 1 for 90 degree rotation else 0, and k+2 is 1 for 270 degree rotation else 0. Alternatively for more rotation options lets make a = input [k+1] and b = input [k+2]. If a = 0 and b = 0 then 0 degrees, if a = 0 and b = 1 then 90 degrees, if a = 1 and b = 0 then 180 degrees if a = 1 and b = 1 then 270 degrees.

From your image set, draw 3 samples. Sample 1 [training set] must be say 50% of the images, sample 2 [validation set] say 30% of the images and sample 3 [out of sample set] 20% of the images.

Use the training images from sample 1 to update the weights. Read in images from sample 1 at random, the rotation must be random, use your image rotator to rotate the image, and compare it to the output of the neural network, calculate the sum of the mean error and perform gradient descent to update the weights. Repeat the process. After every x weight updates randomly select a few images from the validation set and calculate an error value. If the average error value of the validation images are above some threshold then stop, or if the error begins to get worse. Do not use the validation set to update the weights

At the end use the out-of-sample set of images to verify the performance of the neural network.

The process above does not include hyper-parameter optimisation, or any other optimisation techniques, but does describe the basic way of training a NN using supervised learning.

Edit: I assumed that you wanted to NN to learn to rotate the image. After your updates, I assume you want rotated images to increase generalisation of your NN. In which case randomising or shuffling the input batch is preferred. i.e. send your images to Keras to rotate them, get the batch back any order is fine ABab or AaBb and then shuffle them. Pass the shuffled training and validation set to the NN and compare it to the known label for weight updates.


Is there any difference between the convolution operation applied to images and applied to other numerical 2D data? Is there any difference between the convolution operation applied to images and applied to other numerical 2D data?

For example, we have a pretty good CNN model trained on a number of $64 \times 64$ images to detect two classes. On the other hand, we have a number of $64 \times 64$ numerical 2D matrices (which are not considered images), which also have two classes. Can we use the same CNN model to classify the numerical dataset?

Short answer is no. You can't use a model trained for one task to predict on a totally different task. Even if the second task was another image classification task, the CNN would have to be fine tuned for the new data to work.

A couple of things to note...

1) CNNs are good for images due to their nature. It isn't necessary that they'd be good for any 2-dimensional input.

2) By 2D numerical data I'm assuming you don't mean tabular data.


What is the difference between the concepts "known environment" and "deterministic environment"? According to the book "Artificial Intelligence: A Modern Approach", "In a known environment, the outcomes (or outcome probabilities if the environment is stochastic) for all actions are given.", and in a deterministic environment, "the next state of the environment is completely determined by the current state and the action executed by the agent...".

What's the difference between the two terms? Don't they mean the same thing?

What's the difference between the two terms? Don't they mean the same thing?

They mean different things, and can occur in any combination.

A known, deterministic environment

This is an environment where the researcher knows how to calculate all the transitions in advance of observing them, and the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Any classic board game against a fixed opponent (where any situation allows choice of action, the opponent always picks the same choice in the same situation)

An unknown, deterministic environment

This is an environment where the researcher does not have the knowledge to calculate all the transitions in advance of observing them, but any observation the transition from state $s$ given action $a$ is always to the same next state $s'$ with the same reward $r$.

Example: Simple mechanical physics environments, where initial measurements are unknown, imprecise or the researcher does not want to code knowledge of them into the agent. E.g. pole balancing.

A known, stochastic environment

This is an environment where the researcher knows all the rules about transitions, but those rules include transitions with random elements. Transition from state $s$ given action $a$ varies according to some probability function $p(s'|s,a)$ possibly so does the reward $p(r|s,a)$ - sometimes combined into joint probability function $p(r, s'|s,a)$.

Example: Any board game involving dice, e.g. Backgammon.

An unknown, stochastic environment

This is an environment where the researcher does not know all the rules, or can only calculate expected results with difficulty or to a low level of precision. Transition from state $s$ given action $a$ varies according to some unknown probability function $p(r, s'|s,a)$. Learning the transition function may require many samples of $s, a$, and this will be a statistical approximation.

Example: In practice, most complex environments, including real-world physics with friction, fluids, non-perfect measurements.

It is quite common for purpose of experiments to have a simulation where the environment is technically known (because the researcher wrote it, or has access to the code and underlying models), but that agents are written to treat it like this last, more challenging case. Agents that can figure out how to act without prior knowledge of the environment are often of interest.