Can layers of deep neural networks be seen as Hopfield networks? Hopfield networks are able to store a vector and retrieve it starting from a noisy version of it. They do so setting weights in order to minimize the energy function when all neurons are set equal to the vector values, and retrieve the vector using the noisy version of it as input and allowing the net to settle to an energy minimum.

Leaving aside problems like the fact that there is no guarantee that the net will settle in the nearest minimum etc – problems eventually solved with Boltzmann machines and eventually with back-propagation – the breakthrough was they are a starting point for having abstract representations. Two versions of the same document would recall the same state, they would be represented, in the network, by the same state.

As Hopfield himself wrote in his 1982 paper Neural networks and physical systems with emergent collective computational abilities

  The present modeling might then be related to how an entity or Gestalt is remembered or categorized on the basis of inputs representing a collection of its features.

On the other side, the breakthrough of deep learning was the ability to build multiple, hierarchical representations of the input, eventually leading to making AI-practitioners' life easier, simplifying feature engineering. (see e.g. Representation Learning: A Review and New Perspectives, Bengio, Courville, Vincent).

From a conceptual point of view, I believe one can see deep learning as a generalization of Hopfield nets: from one single representation to a hierarchy of representation.

Is that true from a computational/topological point of view as well? Not considering how "simple" Hopfield networks were (2-state neurons, undirected, energy function), can one see each layer of a network as a Hopfield network and the whole process as a sequential extraction of previously memorized Gestalt, and a reorganization of these Gestalt?

Deep Learning is not a generalization of Hopfield networks. Deep Learning is a "generalization" of the neural networks/connectionism field started by Rumelhart and McClelland.
There are two kinds of neural networks:

Directed (Perceptron, MLP, ConvNets, RNNs, etc.)
Undirected (Hopfield Nets, Boltzmann Machines, Energy-based models, etc.)

Any of these can be made deep. As you said, Boltzmann machines are the probabilistic version of Hopfield Networks, and there has been a lot more work on deepifying these models than Hopfield nets: Deep Boltzmann machines, Deep Belief Networks, and deep energy models. Hinton is really the guy you want to read to learn about these models, but you can have a look at this paper which compares the three models.
Not sure about the Gestalt organisation. I guess I'll leave that to your interpretation.


Smaller interest area for images than the size of the image in classification neural networks I have the following binary classification problem, my labeled dataset contains images 96x96 px. Now in every image the interest area is of size 32x32 px in the center of the image, and the images are labeled based on that 32x32 px area. If whatever i am trying to detect is in the outer region of the 32x32 px area the label of that image is not affected.

The problem here is that if i use the whole image when traing, my model will not learn that the interest area is only in the center of the image but on the other hand if i crop the images to be of size 32x32 i am loosing a lot of information which can help the model to train on. I found out that i am getting the best results if i crop the images to be of size 64x64 (kind of a trade of).

Now going to the test set, for the test set it doesn't make sense to use this trade of because the model is not learning anything anyways so i would rather crop the test images to 32x32 but then the test set and train set sizes are not the same.

Has anyone came across this problem before? Can i just pad the test images to the size of the train images? is this a good way to go?

"The problem here is that if i use the whole image when training, my model will not learn that the interest area is only in the center of the image" Yes, it will, if you have enough samples and they are all classified based on this area only.

"but on the other hand if i crop the images to be of size 32x32 i am loosing a lot of information which can help the model to train on" . I don't understand what information you are loosing if you only care about the center 32x32 area.

In the end, you need to ask yourself how will the test input look like and work based on that. If you only care about an area of 32x32 which is always is (and will be) at the center of all images, ignore everything and train and evaluate based just on that area (why add extra unnecessary calculations to the model?). If the size and location of the area might change, don't crop the image.

Bottom line, the test set and the train set need to be of the nature (shape) for the model to work properly.


Why is Information Filter called Information Filter? We all know Information Filter is a dual representation of Kalman Filter. The main difference between Information Filter and Kalman Filter is the way the Gaussian belief is represented. In Kalman Filter, the Gaussian belief is represented by their moments, whereas in Information Filter, the Gaussian belief is represented by the canonical form. The canonical form is comprised of an information matrix ($\Omega$) and an information vector ($\xi$).
As we all know, the name Kalman Filter is after Rudolf E. Kálmán, one of the primary developers of this theory.
Now I want to know where the name information filter came from? Why it is called information filter? What type of information is attached to this filter? Is there any significance behind the nomenclature?
I have the same question about the information matrix and information vector. Is there any significance behind the nomenclature?
I already read Probabilistic Robotics by Sebastian Thrun. Chapter 3 Gaussian filter, in subsection 3.4 The Information Filter. There are many equations and theories but that does not tell us about the nomenclature.

I am probably not qualified to answer your question, but I will attempt. At the very least I hope to start a discussion about the matter because I am unsure.

The (linear) Kalman filter uses the error Covariance as it's main contributor to the Kalman gain. We usually notice the Error covariance decreases over time which can be thought of as the decrease in uncertainty. In the information filter the inverse is true. Our "information matrix" increases with more measurements, because our information increases.

Additionally the measurement update of the information state vector y resembles the way we simply add information. Since

y(i+1) = y(i) + i(i+1)

here we are simply adding information.

This is how I reason and remember the Information filter. I'm not sure where the name came from.

Additionally:
Arthur Mutambara mentions: "The information filter essentially tracks information about the states and not the states them selves". This may be more correct and concise.


How to model inhibitory synapses in the artificial neuron? In the brain, some synapses are stimulating and some inhibiting. In the case of artificial neural networks, ReLU erases that property, since in the brain inhibition doesn't correspond to a 0 output, but, more precisely, to a negative input.

In the brain, the positive and negative potential is summed up, and, if it passed the threshold, the neuron fires.

There are 2 main non-linearities which came to my mind in the biological unit:

potential change is more exponential than linear: small amount of ion channels is sufficient to start a chain-reaction of other channels activation's - which rapidly change global neuron's potential.
the threshold of the neuron is also non-linear: neuron fires only when the sum of its positive and negative potentials passed given (positive) threshold

So, is there any idea how to implement negative input to the artificial neural network?

I gave examples of non-linearities in biological neurons because the most obvious positive/negative unit is just a linear unit. But, since it doesn't implement non-linearity, we may consider implementing non-linearities somewhere else in the artificial neuron.

Principles of Computational Modelling in Neuroscience by David Sterratt, Bruce Graham, Andrew Gillies and David Willshaw discuss it in Chapter 7 (The synapse) and also in Chapter 8 (Simplified models of neurons). Especially in chapter 8, they discuss how to add excitatory or inhibitory synapses to integrate and fire neuron.

There are various ways to add inhibitory synapse: either substrate voltage, inject negative current.


Can the mean squared error be negative? I'm new to machine learning. I was watching a Prof. Andrew Ng's video about gradient descent from the machine learning online course. It said that we want our cost function (in this case, the mean squared error) to have the minimum value, but that minimum value shown in the graph was not 0. It was a negative number!

How can our cost function, which is mean squared error, have a negative value, given that the square of a real number is always positive? Even if it is possible, don't we want our error to be 0?

In general a cost function can be negative. The more negative, the better of course, because you are measuring a cost the objective is to minimise it.

A standard Mean Squared Error function cannot be negative. The lowest possible value is $0$, when there is no output error from any example input.

  How can our cost function which is mean squared error have a value under 0?

It cannot. You don't link the precise graph or lecture where you saw this, but I would suspect Andrew Ng drew a representative graph for any cost function in order to point out that it would typically have an optimum, minimum value. He may have been talking at the same time about MSE as an example.

Many loss or cost functions are designed with an absolute minimum of $0$ possible for "no error" results. In supervised learning that is often a simple consequence of basing the cost on the difference between the model outputs and desired outputs. So in supervised learning problems of regression and classification, you will rarely see a negative cost function value. But there is no absolute rule against negative costs in principle.


classification of unseen classes of image in open set classification I have a scanned image, and they need to be classified in one of the pre-defined image classes, so that it can be sorted. However, the problem is the open nature of the classes. At testing time, new classes of scanned images can be added and the model should not only classify them as unseen (open set image recognition), but it should be able to tell in which new class it should belong (not able to figure out the implementation for this.)
So, I am thinking that the below option can work for the classification of unseen classes

Zero-shot learning: Once the image is classified as unseen, we can then apply zero-shot learning to find its respective class for sorting.

Template matching: Match the test image of unseen classes with all available class images, and, once we have a match, we can do sorting of images.

Meta learning-based approach: I am not sure how to implement this, suggestions are much appreciated.

Note: I already tried the classical computer vision approach, but it's not working out. So, more open for neural net-based approach.
Is my approach to solving the problem correct? If possible, suggest some alternative to find the corresponding match/classification of the unseen class image. As I could think of these 2 alternative solutions only.

I have one possible solution for you inspired by real-time face recognition system. It is a similar case to you.

Create embeddings for each class i.e. person or in your case, a class. (Using Siamese network with ArcFace loss)
When a new image comes, take L1 or cosine distance with the embeddings. You will need a threshold above which you create a embedding for the new guy and give it a new class.

One issue you will think at this point of time is how do you do that when have billions of data points. The search for the matching embeddings can then be done using Hierarchical Navigable Small World graphs (see https://github.com/nmslib/hnswlib).
This solution is loosely connected to the plausible techniques you have mentioned here. And if you still  want something in that regard, I shall add the keyword Active Learning.


How to reduce variance of the model loss during training? I know that stochastic gradient descent always gives different results. What are the best practices to reduce this variance today?
I tried to predict simple function with two different approaches and every time I train them I see very different results.

Input data:

def plot(model_out):
  fig, ax = plt.subplots()
  ax.grid(True, which='both')
  ax.axhline(y=0, color='k', linewidth=1)
  ax.axvline(x=0, color='k', linewidth=1)

  ax.plot(x_line, y_line, c='g', linewidth=1)
  ax.scatter(inputs, targets, c='b', s=8)
  ax.scatter(inputs, model_out, c='r', s=8)

a = 5.0; b = 3.0; x_left, x_right = -16., 16.
NUM_EXAMPLES = 200
noise   = tf.random.normal((NUM_EXAMPLES,1))

inputs  = tf.random.uniform((NUM_EXAMPLES,1), x_left, x_right)
targets = a * tf.sin(inputs) + b + noise
x_line  = tf.linspace(x_left, x_right, 500)
y_line  = a * tf.sin(x_line) + b

Keras training:

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

model.compile(loss='mse', optimizer=tf.keras.optimizers.Adam(0.01))
model.fit(inputs, targets, batch_size=200, epochs=2000, verbose=0)

print(model.evaluate(inputs, targets, verbose=0))
plot(model.predict(inputs))

Manual training:

model = tf.keras.Sequential()
model.add(tf.keras.layers.Dense(50, activation='relu', input_shape=(1,)))
model.add(tf.keras.layers.Dense(50, activation='relu'))
model.add(tf.keras.layers.Dense(1))

optimizer = tf.keras.optimizers.Adam(0.01)

@tf.function
def train_step(inpt, targ):
  with tf.GradientTape() as g:
    model_out = model(inpt)
    model_loss = tf.reduce_mean(tf.square(tf.math.subtract(targ, model_out)))

  gradients = g.gradient(model_loss, model.trainable_variables)
  optimizer.apply_gradients(zip(gradients, model.trainable_variables))
  return model_loss

train_ds = tf.data.Dataset.from_tensor_slices((inputs, targets))
train_ds = train_ds.repeat(2000).batch(200)

def train(train_ds):
  for inpt, targ in train_ds:
    model_loss = train_step(inpt, targ)
  tf.print(model_loss)

train(train_ds)
plot(tf.squeeze(model(inputs)))

There are a few things you can play with:

Try reducing the learning rate, or increasing decay.
Try using regularization(L1/L2 or dropout)
Try using momentum(your model may be stuck in a local minima)
Adjust other hyperparams(nodes, layers, batch size, etc.)

Unless you have some knowledge about the specific cause of high loss variance, the above steps in some amount should get you where you need to go.


Does L1/L2 Regularization help reach an optimum result faster? I understand that L1 and L2 regularization helps to prevent overfitting. My question is then, does that mean they also help a neural network learn faster as a result?
The way I'm thinking is that since the regularization techniques reduce weights (to 0 or close to 0 depending on whether it's L1 or L2) that are not important to the neural network, this would, in turn, result in "better values" for the output neurons right? Or perhaps I am completely wrong.
For example, suppose I have a neural network that is to train a snake to move around a NxN environment. With regularization, the snake will learn faster in terms of survive longer in the game?

I am not aware of any empirical results regarding this question.
But in theory, adding a regularization term shall make the learning task actually even harder, since there is suddenly a second loss term that the network has to be optimized for, which is not even directly related to achieving the original task of fitting the model to the data.
It is true that the regularization term will try drive as many of the weights towards low values as it can. But, at the same time, the other loss term (computed on the original optimization criterion) will try to drive many of the weights to larger values unequal 0 in order to achieve the original training task. (If that wasn't the case, you would be good to go without regularization in the first place.)
Having these competing interests shall then make achieving the original task harder, which I would in theory expect to be more time consuming than, alternatively, allowing the model overfit until a certain admissible error is reached.
After all, the goal of regularization is to improve generalization of a learned model (i.e. to prevent overfitting, as you said), which is about the quality of the outcome and not about how quickly you get to the desired outcome.


Building an AI that generates text by itself Now I know this might break some StackExchange rules and I am definitely open for taking the thread down if it does!
I am trying to build an AI that can write it's own book and I have no idea where to start or what are the appropriate algorithms and approaches to go with.
How should I start and what do I exactly need for such a project?

There have been many methods proposed for text generating, but recurrent network dominates natural language processing with a key component: the perception of time.
Many networks have been tried for text generation, with notable examples such as Markov chain. However RNN have been proven to work the best and is dominating the field of language modelling (text generation).
How text generation works
A neural network that generates text is commonly called a language model. It is trained on large amount of text with labels being the next token. The text generation process uses several random token as the starting phrase and then the network predicts the rest. However the network does not just predicts the most probable word, instead it randomly chooses one of the few most probable token, hence the generating part.
Why RNN work best on language modelling
RNN have a perception of time built into the architecture of teh network. LSTM, a popular RNN variant used, is composed of "memory units" that "remembers" past text, thus the "time" part. RNN process input according to the sequence of time, so the network can naturally understand time, thus the superior performance compared to other networks.
Architecture of language model
A language model consists of the encoder and the decoder. The encoder compresses word one-hot representation to a smaller sized vector representation. The smaller sized representation is then passed through the decoder, which maps the encoding to the words one hot vectors again.
State of the art results for language modelling
Language modelling is an actively researched field in the AI community, and recently the model GPT-2 have achieved a breakthrough in language modelling accuracy, producing almost human like text with a special component added, the attention layer. Attention basically maps the "memory states" of the encoder and feed it as input to the decoder. The data teh model is trained on is also very large, with over 20GB of web scraped data from sites like Reddit.
Limits of language modelling
One limit of language modelling is the size of generated text. As GPU don't have unlimited memory, language model usually limits the input token size to a specific number, padding or trimming to this number. The number is usually 500-1000, which includes a paragraph or two, but not an entire book. You can only generate short paragraphs and essay with language modelling. For long text it is much harder.
Resources to help you get started
GPT-2 open AI blog: https://openai.com/blog/better-language-models/
GPT-2 online interactive site for text generation: https://talktotransformer.com/
How to train and fine tune GPT-2 in python: https://minimaxir.com/2019/09/howto-gpt2/
Hope I can help you


How to make chatbot using NLP like Dialogflow? I want to apply the concept that exists in the Dialogflow API in my e-commerce website.

I get some references in this regard :

Tokenization
Part Of Speech
Named Entity Recognition
Rule based

I just saw that I just didn't understand how to implement it on the website.
so I still don't know how the truth is.

please give me a method or explanation that can help me create a chatbot for ecommerce that can give action when a user asks for a product and wants to place an order or something else.

Please give me some explanation or method or references :(

You are right that you will have to implement the above steps before building any chat bot, this is because your computer doesn't understand text like you and I do. Therefore, text has to be pre-processed(converted) into a format that the computer can work with Tokenization, Parts of Speech Tagging and Named Entity Recognition are such pre-processing techniques.

Rule based means when you create rules for deciding how should your chat bot behave in a given situation.

You can have a look at this blog which teaches how to deploy a chat bot using DialogFlow on Slack - https://www.analyticsvidhya.com/blog/2018/03/how-to-build-an-intelligent-chatbot-for-slack-using-dialogflow-api/


How to source training data in ML for information security? A company entrusts a Data Scientist with the mission of processing and valuing data for the research or treatment of events related to traces of computer attacks. I was wondering how would he get the train data.
I guess he would need to exploit the logs of the different devices of the clients and use statistical, Machine Learning and visualization techniques in order to bring a better understanding of the attacks in progress and to identify the weak signals of attacks... But how would he get labelled data?
He might get the logs of attacks received before, but that might not have the same signature with the attacks that are going to come later? So it might be difficult to create a reliable product?

You have implicitly assumed that supervised learning is being used, given the assumption that labels are needed. But this might lead to the following potential problems:

Log file data tends to be huge, and it may be infeasible to label due to the time/expertise required;
Then there's the class imbalance problem, in that attack examples are far far rarer than "normal" behaviour, and this can mess up supervised models during both learning and evaluation stages;
And finally even if the data is labelled, a supervised model is unlikely to be useful in detecting completely novel attacks because it has not been trained to recognise these.

I think a far easy way to approach these kinds of problem would be unsupervised learning: model normal patterns and behaviours in the logs, and then flag any deviations from normality. It may be an attack or it may be new normal behaviour. In the latter case, the model can be updated. There are various approaches here such as clustering, outlier detection and possibly self-supervised learning that might be useful. Dimensionality techniques might also be useful to visualise clusters of "normal" behaviour that can be compared to abnormal patterns.


Is my understanding of the value function, Q function, policy, reward and return correct? I'm a beginner in the RL field, and I would like to check that my understanding of certain RL concepts.

Value function: How good it is to be in a state S following policy π.

So, the value functions here are 0.3 and 0.9

Q function(also called state-action value, or just action value): How good it is to be in a state S and perform action A while following policy π. It uses reward to measure the state-action value

So, the state-action values here are 0.03,0.02,0.5 and 0.9

Q value: The overall expected rewards after performing action A in state S, and continuing with policy π until the end of the episode. So, essentially I can only calculate the Q value if I know all the state-action values of the actions I will be taking in the single episode.(Because the Q value takes into account the actions after the current action A, till the end of the episode, following policy π)

Reward: The metric used to tell the agent how good/bad it's action was. It is a constant value.
For e.g

 1. Fall in pond --> -1
 2. On stone path --> +1
 3. Reach home--> +10

Return: The sum of rewards in a single episode

Policy π: A set of specific instructions an agent will follow in an episode. For example, the policy will look like:

In state 1, take action 3 ( which takes me to state 2)

In state 2, take action 2 ( which takes me to state 3)

In state 3, take action 1 ( Which takes me to state 4)

In state 4, take action 2 ( Which takes me to terminal state)

1 episode completed

And my policy will keep updating each episode to get the best return

Value function: How good it is to be in a state $s$ following policy $\pi$.

There are different value functions. There's the state value function, often denoted as $v(s)$ (or $V(s)$), so it's a function of only one variable, i.e. $s$ (a state). There's the state-action value function $q(s, a)$ (or $Q(s, a$)). A value function is a function, so it's not a number or a vector, or whatever. It's a function, so it maps inputs to outputs. In the first case, it maps states to real numbers. In the second case, it maps states and actions to real numbers. So, we could denote the state value function as $v : \mathcal{S} \rightarrow \mathbb{R}$ (where $\mathcal{S}$ is the set of states in your environment) and state-action value function as $q : \mathcal{S} \times \mathcal{A}\rightarrow \mathbb{R}$ (where $\mathcal{A}$ is the set of actions and $\times$ means "combination of").

So, your definition of a value function is not quite correct. The value function $v(s)$ doesn't represent "how good it is to be in a state $s$ following a policy $\pi$", but "how good it is to be in a state $s$ AND THEN following policy $\pi$". To emphasize this, you often use the notation $v_{\pi}(s)$ rather than simply $v(s)$.

See What are the value functions used in reinforcement learning? for more details about existing value functions in reinforcement learning. And to see the full definition of the value functions, I suggest you read Sutton and Barto's book.

  Q function (also called state-action value, or just action value): How good it is to be in a state $s$ and perform action $a$ while following policy $\pi$. It uses reward to measure the state-action value

As I said above, the $q$ function is a "value function" too. It's just a different value function than $v$.

Again, the same thing I said for $v$ also applies here, so "how good it is to be in a state $s$ and perform action $a$ while following policy $\pi$" is incorrect for the same reason your definition for $v$ was incorrect. The $q$ function can be defined as "how good it is to be in a state $s$ and take action $a$, AND, AFTER THAT, follow a given policy $\pi$. Again, to emphasize that $q$ is defined in terms of $\pi$, we often use the notation $q_\pi$.

  Reward: The metric used to tell the agent how good/bad it's action was. It is a constant value.

This is roughly correct, but the reward doesn't have to be constant and it depends on your problem. Also, there's also the related notion of "reward function", which is the function that assigns rewards to each action. So, when defining your problem as a Markov decision process, you need to define this reward function. Actually, this is probably the most important function in reinforcement learning (because this is the way you teach the agent to behave).

  Return: The sum of rewards in a single episode

This is roughly correct. However, note that the sum can also be a "weighted sum".

  Policy: A set of specific instructions an agent will follow in an episode.

This is roughly correct, but a policy can also have some randomness in it. For example, if you are in state $s$, your policy could say "always take action $a_i$", but another policy could say "take action $a_i$ with probability $p$ and action $a_j$ with probably $1 - p$. Also, note that the policy is not restricted to an episode. It's a general function that tells the agent how to behave independently of the episode.

(Sorry, I didn't look at your examples. Maybe I will review this answer later to look at your examples too, but the information in this answer should already tell you if your examples are correct or not).


How to make deepfake video without a fancy PC? Is there any way to make deepfake videos without a fancy computer? For example, run the DeepFaceLab on a website so your own computer won't get involved?

Yes. There are services that provide free environment to run jupyter notebooks for research purposes (with GPU included, which is crucial for neural networks) - such as Google Colaboratory and Kaggle Kernels. Although they limit how long your computation may run (12 and 6 hours accordingly), which adds some difficulties to the process, although I think it is possible to bypass these restrictions.


How long it takes to train face recognition deep neural network? (rough estimation) If I use a desktop PC with a GPU, how long it might take to train face recognition deep neural network on let's say dataset of 2.6 million images and 2600 identities? I guess it should depend on various properties (e.g., type of the DNN). But I am just looking for a rough estimation. Is it a matter of hours/days or years?

Thanks!

Training time depends on a lot of parameters. Some of them are:

Size of each image (resolution)
Color/Monochrome image (color image has 3 times data if you consider RGB image)
Like you mentioned on the type of DNN.
No. of layers of DNN.
No. of neurons in each layer.
Total no. of images in the dataset. (2.6 million here)
GPU you are using (you didn't mention which GPU you are using. There are GPUs with wide range of capabilities, to predict the time, you need to know the exact specs of GPU).
Training time also depends on RAM on your machine and also how fast host PC can transfer data to GPU for processing.
Since you mentioned face recognition, i am assuming you are using CNN, but if you again use fully connected network, training time will change and will obviously increase manifold.

Your task of classification and your database is mostly similar to ILSVRC that uses imagenet database.

Making some reasonable assumptions for the parameters you didn't mention, i feel your task is similar to ILSVRC and i am predicting the training time will be a few days.

Below are the links which mention time of training for ILSVRC.

https://mxnet-tqchen.readthedocs.io/en/latest/tutorials/imagenet_full.html

Below are the details of imagenet database for your comparison

https://en.wikipedia.org/wiki/ImageNet


Extending a neural network to classify new objects Suppose a model M classifies apples and oranges. Can M be extended to classify a third class of objects, e.g., pears, such that the new images for 'retraining' only have pears annotated and apples and oranges ignored? That is, since M already classifies apples and oranges, can the old weights be somehow preserved and let the retraining focus specifically on learning about pears?

Methods such as fine-tuning and learning without forgetting seem to require all objects in the new images annotated though.

Yes this is standard transfer learning. Using a trained model, we can freeze the first N hidden layers of a classifier, except for the last few. This will allow our previous relevant training to be retained whilst also being able learn new features and target new classes.
We will then initialize a new output layer that works our new context(i.e sigmoid, 1 node for binary classifier). Everything is now set to resume training on our new y_targets.
Take a look at this link for some more info on transfer learning.
If you want no perturbance to your past learning, I would recommend freezing all previous hidden layers and then tacking some additional ones on.


How can I prevent the CNN from classifying a new input into one of the existing labels (it was trained with) when the input has a new different label? I'm trying to perform image classification with a CNN. In my case, the inputs are the covers of 9 books, so there are 9 labels. I am using TensorFlow's Keras.
If I pass a new input (that has a label different than one of the 9 labels the CNN was trained with), it will be classified as one of the 9 books, even though it's not a book (but it's e.g. a wall, sofa, house, etc.). I want to avoid this. I want the model to first classify whether there is a book in the image and then classify the book in 9 classes. How could I achieve this?

You can introduce another class to your network - "not a book". After that, you will need to add new data to your dataset, random images that do not contain books to classify and train your network on that data. So when your network won't see a book it will output high probability for "not a book" class, if an image with a book will be shown to the network probability of the "not a book" class should be low.


How agent's reasoning skills can improve its reinforcement learning? Agent can have reasoning skills (prediction, taking calculated guesses, etc.) and those skills can help reinforcement learning of this agent. Of course, reinforcement learning itself can help to develop reasoning skills. Are there research that explores this impact of reasoning and consciousness on the effectivenes of reinforcement learning. Or maybe people just sit and wait such skills to emerge during reinforcement learning?

It sounds like you are describing a synthesis of two competing ways to solve the MDP problem.

In reinforcement learning, we solve the MDP problem by having the agent move around its environment, observe rewards and transitions in response to the actions it takes, and build a model of the relationship between actions and rewards that allows it to maximize rewards.

An older approach is to give the agent facts about the world that can be encoded as logical rules. The agent then uses unification to reason about the consequences of actions within that framework of rules. The agent then takes actions that maximize the rewards it can expect, given the rules and the information at hand. A problem with this approach was that it did not work well in problem domains with probabilistic rules (i.e. X usually happens when action Y is taken).

A hybrid approach, somewhere between these two is the use of Value Iteration or Policy Iteration methods. These are so-called "model-based" reinforcement learning algorithms (although, I would tend to say that makes them something other than true reinforcement learning...). Like the older logic-based approaches, they start by writing down a series of rules that fully describe how things happen in the world, and then derive the logical consequences of those rules to compute the best actions the agent could take. Like reinforcement learning however, they are able to account for probabilistic rules and probabilistic rewards. If you had an exact description of a game of chance, you could write it down as an MDP, and then solve it exactly using these techniques.

Importantly, value & policy iteration methods are not feasible if the state and action spaces are very large (as they often are), and are really not feasible if the MDP is not known exactly (i.e. if you don't know all the rules of the game in advance). That's where reinforcement learning shines.


Can a sentence have different parse trees? I just read about the concept of a parse tree.
In my understanding, a valid parse tree of a sentence needs to be validated by a linguistic expert. So, I concluded, a sentence only has one parse tree.
But, is that correct? Is it possible a sentence has more than one valid parse tree (e.g. constituency-based)?

But, is that correct? Is it possible a sentence has more than one valid parse tree (e.g. constituency-based)?

The fact that a single sequence of words can be parsed in different ways depending on context (or "grounding") is a common basis of miscommunication, misunderstanding, innuendo and jokes.
One classic NLP-related "joke" (around longer than modern AI and NLP) is:

Time flies like an arrow.
Fruit flies like a banana.

There are actually several valid parse trees for even these simple sentences. Which ones come "naturally" will depend on context - anecdotally I only half got the joke when I was younger, because I did not know there were such things as fruit flies, so I was partly confused by literal (but still validly parsed, and somewhat funny) meaning that all fruit can fly about as well as a banana does.
Analysing these kinds of ambiguous sentences leads to the grounding problem - the fact that without some referent for symbols, a grammar is devoid of meaning, even if you know the rules and can construct valid sequences. For instance, the above joke works partly because the nature of time, when referred in a particular way (singular noun, not as a possession or property of another object), leads to a well-known metaphorical reading of the first sentence.
A statistical ML parser could get both sentences correct through training on many relevant examples (or trivially by including the examples themselves with correct parse trees). This has not solved the grounding problem, but may be of practical use for any machine required to handle natural language input and map it to some task.
I did check a while ago though, and most Parts Of Speech taggers in Pythons NLTK get both sentences wrong - I suspect because resolving sentences like those above and AI "getting language jokes" is not a high priority compared to more practical uses for chatbots/summarisers, etc.


Is the emergence of the field "psychology of artificial intelligence" necessary? Is the emergence of a subfield of psychology, "psychology of artificial intelligence", possible and necessary? If so, what questions it will address and will try to solve?

Possibly, it could also emerge as a new branch of engineering psychology, due to the fact that engineering psychology handles the realms of interactions between humans and machines, as well as human factors in general.

And by trying to answer my own question, and how I see it, yes - there is some requirement like it stated in this particular article: https://medium.com/womeninai/psychology-of-artificial-intelligence-ca0f0a9f3d7c

Because yes - there are multiple occurrences of an AI being biased, hateful, racist, sexist, etc.

Also, humans themselves tend to act towards an AI in the insolent ways, so the human factor also needed to be accounted for.

So yeah - I think it will be a branch, or an extension of an engineering psychology, which exactly addresses primarily a human factor and relationships between the human and a machine.

But other than that it will address the factors of how should it be built, how we can expect it to behave, and how should it behave, because it should follow the strict moral code, to be not the threat to us.


Can alpha-beta pruning be used for applications apart from games? Can alpha-beta pruning/ minimax be used for systems apart from games? Like for selecting the right customer for a product, etc. (the typical data science problems)? I have seen people do it, but can't understand how. Can someone help me understand that?

Can I do something like if - find two criteria on which customers can buy product depends on like gender and age. Find the probability for all the customers depending on age and gender if they can buy it.

like if there are 3 customers - there probability to buy a product on the basis of their age and gender is - Customer 1 - (20%, 30%), Customer 2 - (30%, 60%), Customer 3 - (40%, 20%). here the x and y represents - (probability based on age, probability based on gender ). Probability is probability to buy the product.

For minimax, will it be correct if one player(max) tries to select the customer on basis of gender and other player(min) on basis of age. so, one can be max and one can be min.

Dont know if this correct or not, but just a idea.

Thinking about this more, the answer is in fact yes, but not for the application you mention.

You cannot use alpha-beta pruning to learn a model to predict customer outcomes, because it is only useful for domains where you are concerned about an adversary. In finding a customer model, there is no reason to worry about someone coming in and forcing you to make bad decisions about the optimization of the model. Consequentially, there is no reason to use minimax search, and thus, to use alpha-beta pruning.

There are applications other than (video) games where you could use these techniques though. For example, there are security games. In these "games" we want to use AI to find a strategy to protect an airport. It is reasonable to try and design our model under the assumption that someone else wants to break it. You could use Alpha-Beta pruning here (although in practice, more sophisticated algorithms are used).


How sigmoid funtion helps us in reducing error in neural networks? As in sigmoid function when input x is very large or very small the curve is flat that means low gradient descent but when it is in between the slope is more so,
My question is how this thing helps us in neural network.

There are several nice things about using the logistic function and a few drawbacks as well.  I'm not going to discuss all of them, but here is a small synopsis.
Nice Properties

For networks which calculate probabilities, it brings the input features into a nice 0 to 1 range.
Since the logistic function maps inputs to the range (0,1), it is commonly used as an activation function in neural networks which require some sort of classification.  For example, if we had a convolution neural network (CNN) that classified if an image given to it was a dog or not, it could have an output neuron that returned 0 if the image was not a dog or 1 if it was.  When the CNN is given an image with inputs of varying dimensions and ranges, it's more intuitive to deal with the input features in the realm of 0 to 1 instead of 0 to 255 or whatever input format you're dealing with.
It is smooth and differentiable.
When performing back propagation, it's much easier to deal with differential functions.  Otherwise you'd have to approximate the gradient which may lead to unfavorable results.
A Vanishing Issue
At extreme values, the gradient vanishes.
The OP was also hinting that at very positive and very negative values of X, derivative of the logistic function could become close to 0.  For large input values this may hinder learning for extreme values.  If you think of gradient decent, the jump that gradient descent makes is only as strong as its magnitude of the gradient that it requires.  Analytically speaking, this also hurts back propagation because learning is calculated using the chain rule, which ends up multiplying several gradients together.


Ensemble models - XGboost I am building 2 models using XGboost, one with x number of parameters and the other with y number of parameters of the data set.

It is a classification problem.
A yes-yes, no-no case is easy, but what should I do when one model predicts a yes and the other model predicts a no ?

Model A with x parameters has an accuracy of 82% and model B with y parameters has accuracy of 79%.

Without any additional information, lean towards the vote of the best performing classifier when it comes to ties.

However, as others have stated already, it is best to analyze the performances in more detail (e.g. confusion matrices).

For instance, it could be that model B almost always classifies class X correctly (hardly any false positives). In that case, you could lean towards the prediction of model B if it predicts class X and model A does not. In other words; you could weigh the votes of the models based on how well they did in similar, previous predictions.


Is it possible for 'unsupervised learning' model to recognize features on unlabelled images? Is it possible for unsupervised learning to learn about high-level, class-specific features given only unlabelled images? For example detecting human or animal faces? If so, how?

I am assuming each image contains a single object.

It is possible, however, it is not as easy as you might think. Firstly, you need extract as many features as possible: original image, LBP, SIFT, moments, contour descriptors to name a few. Than concatenate these features into a single feature vector. After this step, use clustering. You will need a lot of samples to compensate for the number of features. After clustering, use a correlation method to find which features are related to each cluster.

If you need features to classify within a cluster, you could do a second clustering with full set of features and apply the same method. The features that are selected for a cluster will not be suitable to classify within the cluster.


Reinforcement learning: How to deal with illegal actions? I'm a beginner of RL and currently trying to make DQN agent that can act optimally in a simple situation.

In the situation agent should decide at what rate to charge or discharge the electrical battery, which is equivalent to buying the electrical energy or selling it, for making money by means of arbitrage. So the action space is for example [-6, -4, -2, 0, 2, 4, 6]kW. The negative numbers mean discharging, and the positive numbers mean charging.

In a case that battery is empty, discharging actions(-6, -4, -2) should be forbidden.
Otherwise in a case that battery is fully charged, charging actions(2, 4, 6) should be forbidden.

To deal with this issue, I tried two approaches:

In every step, renewing the action space, which means masking the forbidden action.
Give extreme penalties for selecting forbidden actions (in my case the penalty was -9999)

But none of them worked.

For the first approach, the training curve (the cumulative rewards) didn't converge.

For the second approach, the training curve converged, but the charging/discharging results are not reasonable (almost random results).
I think in second approach, a lot of forbidden actions are selected randomly by the epsilon-greedy policy, and these samples are stored in experience memory, which negatively affect the result.

for example:

The state is defined as [p_t, e_t] where p_t is the market price for selling (discharging) the battery, and e_t is the amount of energy left in the battery.

When state = [p_t, e_t = 0], and discharging action (-6), which is forbidden action in this state, is selected, the next state is [p_t, e_t = -6]. And then the next action (2) is selected, then the next state is [p_t, e_t = -4] and so on.

In this case the < s, a, r, s' > samples  are:

< [p_t, 0], -6, -9999, [p_t+1, -6] >

< [p_t, -6], 2, -9999, [p_t+1, -4] > ...

These are not expected to be stored in the experience memory because they are not desired samples (e_t should be more than zero). I think this is why desired results didn't come out.

So what should I do? Please help.

In my project I also had the problem that the action space is not the same for every state of the environment. I do not like the approach to penalty forbidden actions with a high negative reward since it feels a bit like cheating. However it might work, I just haven't tried it.

The approach I used, which you could apply as well, is to integrate an additional function into your action space. This function would map an action to a specific amount of kW. Thereby, depending on the current state, the function maps the action to the amount of kW to charge or discharge your battery with. This has the advantage that you do not have to deal with illegal actions.

This could be applied as follows:
Instead of defining for every action the amount to charge / discharge your battery with, you create a set of functions that defines the respective amount. Here an example with five actions:

Action: Discharge the battery entirely
Action. Discharge the battery so that half of its capacity remains,
otherwise do nothing
Action: do nothing
Action: charge the batter to half its capacity, otherwise do nothing
Action: charge the battery to its maximum capacity


Has any research been done on DNN Music? DNNs are typically used to classify things (of course) but can we let them go wild with sounds and then tell them if we think it sounds good or not? I'd like to think after a training class has been made (perhaps comparing the output to an existing song) we could get an NN that has a basic concept of music.

Timing would be an issue; I'm not sure how feasible this is. A strongly weighted input attached to all hidden layers perhaps? Use it as the bias?

Is this even slightly feasible?

The first thing is to define what is a «good» and a «bad» sound. This is an extremely tricky issue, since the networks need numeric inputs. And music is whole bunch of numbers.

I know from people doing research in identifying how similar two sounds are, and imitation, say: you hear a sound and try to make another that sounds like it. Like when you hum a song or similar. That is by no means easy. These guys are using something similar to feature extraction, with Fourier transforms and energy and such things. They feed the networks with the (selected) features and... Train.

Now, to return to your original question: *What do you present as target during training?* You can present different types of music as categories and classify (I couldn't help but think on this research with fish). Or you define categories of music you like and see if the network can classify them ;)

One basic decision here is how long you get a piece of sound. Since it is needed to analyse frequency, this is a key issue. Since you talked about DNN, I was wondering if you wanted to do it online, as a stream, in which case I don't have the slightest idea where to begin, other than do it after a little while.

Other idea: I remember a little sketch in this series about a researcher that makes use of the relations between peaks in the Fourier spectrum in order to differentiate noise from music.


How should we interpret all the different metrics in reinforcement learning? I'm trying to train some deep RL agents using policy gradient methods like AC and PPO. While training, I have a ton of different metrics being monitored.
I understand that the ultimate goal is to maximize the reward or return per episode.
But there are a ton of other metrics that I don't understand what they are used for.
In particular, how should one interpret the mean and standard deviation curves of the policy loss, value, value loss, entropy, and reward/return over time while training?
What does it mean when these values increase or decrease over time? Given these curves, how would one decide how to tune hyperparameters, see where the training is succeeding and failing, and the like?

As you said, generally the most important one is reward per episode. If this isn't increasing overall, there's a problem (of course this metric can fluctuate, I mean to say that macroscopically it should increase).
Policy loss (I assume you mean the "actor loss"?) is generally harder to interpret. You should think of this more as a source of gradients and not necessarily a good indicator of how well your agent is performing.
I'm not really sure why you'd be monitoring the value during training. Value loss, however, is basically equivalent to value loss in value based methods like Q-learning, for example. So this one should be decreasing overall. Otherwise the baselines you compute to reduce variance in the policy gradient will either be less effective, or even harmful.
Entropy is a nice quantity to measure, because it's a good indicator of how much your agent is exploring. If you see that your agent is not achieving high returns and the entropy is really low, this means that your policy has converged to a suboptimal one. If the entropy is really high, this means the agent is acting fairly randomly (so it's basically exploring a lot). Ideally the entropy should decrease over time, so your policy becomes more deterministic (less exploration) as it reaches an optimum.


Backpropagation: Chain Rule to the Third Last Layer I'm trying to solve dLoss/dW1. The network is as in picture below with identity activation at all neurons:

Solving dLoss/dW7 is simple as there's only 1 way to output:

$Delta = Out-Y$

$Loss = abs(Delta)$

The case when Delta>=0, partial derivative of Loss over W7 is:

$\dfrac{dLoss}{dW_7} = \dfrac{dLoss}{dOut} \times \dfrac{dOut}{dH_4} \times \dfrac{dH_4}{dW_7} \\
= \dfrac{d(Out-Y)}{dOut} \times \dfrac{d(H_4W_{13} + H_5W_{14})}{dH_4} \times \dfrac{d(H_1W_7 + H_2W_8 + H_3W_9)}{dW_7} \\
= 1 \times W_{13} \times H_1$

However, when solving dLoss/dW1, the situation is very different, there are 2 chains to W1 through W7 and W10, and now, how should the chain for $\dfrac{dLoss}{dW_1}$ be?

Furthermore, at an arbitrary layer, with all outputs of all layers already calculated plus all gradients of weights on the right side also calculated, what should a formula for $\dfrac{dLoss}{dW}$ be?

I finally solved it out but it's long, it's not only chain rule, it includes quotient rule too. And, this is only the third last layer, once a DNN has more layers then it's more complex.

$\dfrac{dLoss}{dW_1} = \dfrac{d}{dW_1}(Out-Y) = \dfrac{d}{dW_1}Out = \dfrac{d}{dW_1}(H_4W_{13} + H_5W_{14}) \\= \dfrac{d}{dW_1}H_4W_{13} + \dfrac{d}{dW_1}H_5W_{14} \\= W_{13} \times \dfrac{d}{dW_1}H_4 + W_{14} \times \dfrac{d}{dW_1}H_5 \\= W_{13} \times \dfrac{d}{dW_1}(H_1W_7 + H_2W_8 + H_3W_9) + W_{14} \times \dfrac{d}{dW_1}(H_1W_{10} + H_2W_{11} + H_3W_{12}) \\= W_{13} \times \dfrac{d}{dW_1}H_1W_7 + W_{14} \times \dfrac{d}{dW_1}H_1W_{10} \\= W_{13}W_7 \times \dfrac{d}{dW_1}H_1 + W_{14}W_{10} \times \dfrac{d}{dW_1}H_1 \\= (W_{13}W_7 + W_{14}W_{10}) \times \dfrac{d}{dW_1}(X_1W_1 + X_2W_2) \\= (W_{13}W_7 + W_{14}W_{10}) \times X_1$


Which neural networks can be used only for storing and retrieving information? Is there a neural network(NN) system or architecture which can be used for only storing and retrieving information. For example; to store whole Avatar movie in HD format inside a neural network and retrieve(without loss) it from the neural network when needed. I searched the web and came across only LSTM RNN but in my understanding LSTM only stores pattern and not the content itself. If there is no such NN exist can you explain why it so?

Neural networks are usually trained to produce a certain output given a certain input. Often the output is a classification of the input or some other form of input description. Sometimes it is an action in a game and sometimes it is indeed stored data, a memory if you will.

In that case the input is often a part of the stored data, so the NN actually completes the given input. This setup is called autoassociative memory, Hopfield networks are an example for this.

In your example you might give this NN the first frame of the movie Avatar and it would output the complete movie. Unfortunately this would probably be absolutely crazy inefficient.


Is Nassim Taleb right about AI not being able to accurately predict certain types of distributions? So Taleb has two heuristics to generally describe data distributions. One is Mediocristan, which basically means things that are on a Gaussian distribution such as height and/or weight of people.

The other is called Extremistan, which describes a more Pareto like or fat-tailed distribution. An example is wealth distribution, 1% of people own 50% of the wealth or something close to that and so predictability from limited data sets is much harder or even impossible. This is because you can add a single sample to your data set and the consequences are so large that it breaks the model, or has an effect so large that it cancels out any of the benefits from prior accurate predictions. In fact this is how he claims to have made money in the stock market, because everyone else was using bad, Gaussian distribution models to predict the market, which actually would work for a short period of time but when things went wrong, they went really wrong which would cause you to have net losses in the market.

I found this video of Taleb being asked about AI. His claim is that A.I. doesn't work (as well) for things that fall into extremistan.

Is he right? Will some things just be inherently unpredictable even with A.I.?

Here is the video I am referring to https://youtu.be/B2-QCv-hChY?t=43m08s

Yes and no!

There's no inherent reason that machine learning systems can't deal with extreme events. As a simple version, you can learn the parameters of a Weibull distribution, or another extreme value model, from data.

The bigger issue is with known-unknowns vs. unknown-unknowns. If you know that rare events are possible (as with, say, earthquake prediction), you can incorporate that knowledge into the models you develop, and you'll get something that works as well or better than humans in that domain. If you don't know that rare events are possible (as with, say, a stock market crash produced by correlated housing defaults), then your model will reflect that as well.

I tend to think Taleb is being a bit unfair here: AI can't handle these kinds of events precisely because its creators (us) can't handle them! If we knew they were possible, then we could handle them pretty well, and AI could too.


Artificial Intelligence and Automatic Scheduling I do not have background in Artificial Intelligence related algorithms etc. But I have a scheduling problem that needs to be resolved.

For example, employees need to be scheduled for on call. They might be on vacation.

Can you guide as to what algorithms I need to look into, suitable for a beginner?

Scheduling can get quite complicated. It is fundamentally programming to find a solution given constraints. One approach would be to use MiniZinc, and here is a scheduling example. You can use other plain programming approaches such as Python and so on, but Minizinc is highly optimized and might well reward time invested in learning by finding a better solution more quickly.


How to stay a up-to-date researcher in ML/RL community? As a student who wants to work on machine learning, I would like to know how it is possible to start my studies and how to follow it to stay up-to-date. For example, I am willing to work on RL and MAB problems, but there are huge literatures on these topics. Moreover, these topics are studied by researchers from different communities such as AI and ML, Operations Research, Control Engineering, Statistics, etc. And, I think that several papers are published on these topics every week which make it so difficult to follow them.

I would be thankful if someone can suggest a road-map to start studying these topics, follow them and how I should select and study new published papers. Finally, I am willing to know the new trend in RL and MAB problem.

There are some wonderful resources for keeping up to date in the ML community. Here are just a handful that a coworker showed me:

Deep Learning Monitor: this site contains hot and new papers along with tweets that are popularized by the community! You can even checkout RL papers specifically here
arxiv-sanity: this site updates with popular and new papers that make it onto Arxiv
papers with code: this site is wonderful because not only does it link to papers, but it links to their implementation for reproduction or assistance in your own personal projects. They even have a leaderboard and track state of the art (SoTA) on tons of different tasks
DL_twitter loop: You can't forget twitter, given that most researchers use it; this is just a single nice group you may like


Can the rewards be stochastic when the transition model is deterministic? Suppose we have a deterministic environment where knowing $s,a$ determines $s'$. Is it possible to get two different rewards $r\neq r'$ in some state $s_{\text{fixed}}$? Assume that $s_{\text{fixed}}$ is a fixed state I get to after taking the action $a$. Note that we can have situations where in multiple iterations we have: $$(s,a) \to (s_1, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_1) \\ (s,a) \to (s_{\text{fixed}}, r_2) \\ (s,a) \to (s_3, r_3) \\ \vdots$$

My question is, would $r_1 =r_2$?

My question is, would $r_1 =r_2$?

That's usually up to you as the designer of the system.

Usually when you declare that you have "a deterministic environment", you imply that both $s'$ and $r$ are fixed values depending on $(s,a)$. So in your examples, you would expect your observations to also have $r_1 = r_2$

However, it is possible to define a MDP where transition to state $s'$ is deterministic, but $r$ is not. For instance, you could define reward in a game equal to the sum of a number of dice rolled, with better rewards (on average) resulting in more dice. This is still a valid MDP and can be solved using RL techniques.

A real-world example of this might be managing a queue of work, where you want to minimise lead time, but don't know for certain how long each task will take. Your state progression moves deterministically - you have a queue of pending tasks, current tasks and workers, and assigning a task to a worker is completely deterministic. However, you don't know how efficiently tasks will be performed until after they are done, so you don't know the reward perfectly from the assignment (whether you can treat this as random or hidden state is a more complex issue - it is often pragmatic to treat such unknown data as random though).


How to design an ANN to give an answer that includes two different components? As an experiment, I want to teach an ANN to play the game of Nim.

  The normal game is between two players and played with three heaps of any number of objects. The two players alternate taking any number of objects from any single one of the heaps. The goal is to be the last to take an object.

The game is easily solvable and I already wrote a small bot that can play Nim perfectly to provide data sets for supervised learning.

Now I am struggling with the design question, how should I output the solution to a specific board state. The answer always consists of two components:

How many stones to take (a more or less arbitrary integer value)
Which heap to take the stones from (the index of the heap)

What are available design choices in this regard and is there a state-of-the-art design for this type of problem?

Nim is a simple game and it's really simple to build a bot that gives the optimal solution.

  The correct move is to leave an odd number of piles of size 1

So when it comes to training an ANN to play a game, there are some this to keep in mind.

Fixed play area (This is taken care of as there are only 3 stacks)
Providing input to the model so it knows it's current state (You can use a simple array of integers [3, 4, 1] denoting the current state of the stacks)
Providing feedback. An important step to guide the network in the right direction (You already have an optimal bot to do this)

Now that you can easily cover all the requirements, it's pretty simple to teach the model.

Input - Current state of the model [3, 4, 5]
Output - Move that model will make [2, 0, 0]
Final output - Here you will have to add another layer with a custom function.

This is the important part, to direct the model. Check if the move made is GOOD, BAD or ILLEGAL and assign a custom output to the same. For example

GOOD return1 if it matches one of the optimal moves.
BAD return 0 if it's not the optimal move and
ILLEGAL return -1 if the move is not allowed.

Another thing to keep in mind is that this might require a slightly larger network as it as to learn complex functions that are not linearly separable.

You don't need to train that model by playing it against itself as you already have the optimal solution. Playing against itself is required only when you are trying to achieve something that better than the current best agent.

Check this out, it's pretty interesting Neural network to play snake


What is the difference between artificial intelligence and robots? What is the difference between artificial intelligence and robots?

Although there are several definitions of "robot", an essential feature of everything called "robot" is that it is capable of movement. This does not necessarily mean displacement; a robot arm in a factory also moves.

There is a single exception to this rule,  which is bot-programs like chatbots; I will discuss them later.

Artificial Intelligence does not need to move; a chess program can be argued to be an AI, but does not move.

A robot can actually have AI; one of the definitions of robot is that it is a system, capable of autonomous movement. In order to be autonomous, to be able to make decisions of its own, a certain amount of AI may be necessary.

There is one class of "robots" that does not move, and does not even have physical presence; bot programs, like chatbots, that operate inside systems. I do not consider them robots, because they are not physical devices operating in the real world. A chatbot can be an AI, however - a good chatbot may have some natural language processing to interact with humans in a way that humans find natural.

To summarize; an AI can exist purely in software. But to be a robot, there must be a moving physical component in the real world.


What are examples of simple problems and applications that can be solved with AI techniques? What are examples of simple problems and applications that can be solved with AI techniques, for a beginner who is trying to make use of his basic programming skills into AI at the beginning level?

This is fairly boilerplate advice, but, since you're brand new to AI, I'd personally suggest writing a classical Tic-Tac-Toe AI, ideally using minimax.

I suggest this because minimax is fundamental to AI, and there are many webpages devoted to this subject, such as How to make your Tic Tac Toe game unbeatable by using the minimax algorithm and Tic Tac Toe: Understanding the Minimax Algorithm.  (Google search for "Tic-tac-toe" and "minimax" will yield a plethora of other sites.  I'd also recommend looking at this minimax page from Stanford: "Strategies and Tactics for Intelligent Search".)

I recommend this approach as a good basic primer.  The real cutting-edge work is being done in Machine Learning and Neural Networks, and for that reason, it's probably more important than ever to have some basic grounding in classical AI before you start dipping your toe in that pond.


What is the purpose of the batch size in neural networks? Why is a batch size needed to update the weights of a neural network?

According to that Youtube Video from 3B1B, the weights are updated by calculating the error between expectation and outcome of the neural net. Based on that, the chain rule is applied to calculate the new weights.

Following that logic, why would I pass a complete batch through the net? The first entries wouldn't have an impact on the weighting.

Do I need to define a batch size when I use backpropagation?

tl;dr: A batch size is the number of samples a network sees before updating its gradients. This number can range from a single sample to the whole training set. Empirically, there is a sweet spot in the range 1 to a few hundreds, where people experience the fastest training speeds. Check this article for more details.

A more detailed explanation...

If you have a small enough number of samples, you can let the network see all of the samples before updating its weights; this is called Gradient Descent. The benefit from this is that you guarantee that the weights will be updated in the direction that reduces the training loss for the whole dataset. The downside is that it is computationally expensive and in most cases infeasible for deep neural nets.

What is done in practice is that the network sees only a batch of the training data, instead of the whole dataset, before updating its weights. However, this technique does not guarantee that the network updates its weights in a way that will reduce the dataset's training loss; instead it reduces the batch's training loss, which might not the same thing. This adds noise to the training process, which can in some cases be a good thing, but requires the network to take too many steps to converge (this isn't a problem since each step is much faster).

What you're saying is essentially training the network each time on a single sample. This is formally called Stochastic Gradient Descent, however the term is used more broadly to include any case where the network is trained on a subset of the whole training set. The problem with this approach is that it adds too much noise to the training process, causing it to require a lot more steps to actually converge.


Why would neural network dream scenes mirror the hallucinations people experience when they're tripping? In DeepDream wikipedia page it's suggested that a dreamlike images created by a convolutional neural network may be related to how visual cortex works in humans when they're tripping.

  The imagery to LSD- and psilocybin-induced hallucinations is suggestive of a functional resemblance between artificial neural networks and particular layers of the visual cortex.

How this is even possible?

How exactly convolutional neural networks have anything to do with human visual cortex?

The similarity of artificial neural networks and the human visual cortex goes very deep, and in many ways the human visual cortex was the inspiration for the techniques we use for the design and implementation of ANNs designed for image recognition. So in that direction, the similarity seems obvious to me.

The reverse direction, though, is a question about how the human mind works under the influence of LSD, which you'll probably get a better answer asking about in the biology or cognitive science stack exchange sites.

Some brief details to add to the answer, though: the human visual cortex is arranged in layers that correspond to increasing layers of abstraction. In the eyes themselves, photons are detected by light-sensitive cells and added together to make what are essentially the color elements of pixels. Those are then routed to another layer which does something like edge detection, and then the next layer does something like shape detection, and so on up to higher level concepts like "a cat's face.' If LSD lowers the activation threshold for those neurons, or makes them more excitable, then more things will be interpreted as having the higher level concept (and so a patch of rough texture may have a face jump out of it, for example).

The way that CNN "deep dreaming' works is that the base image is amplified. That is, to make a particular patch look more like a dog, the shapes are nudged to be more dog-like, and the shapes nudge the edges, and the edges nudge the pixels.


Can exogenous variables be state features in reinforcement learning? I have a question about state representation of Q-learning or DQN algorithm.
I'm still a beginner of RL, so I'm not sure that is it suitable to take exogenous variables as state features.

For example, in my current project, deciding to charge/discharge an electric vehicle actions according to the real-time fluctuating electricity prices, I'm wondering if the past n-step prices or hours can be considered as state features.

Because both the prices and the hour are just given information in every time step rather than being dependent to the charging/discharging actions, I'm suspicious about whether they can are theoretically qualified to be state features or not.

If they are not qualified, could someone give me a reference or something that I can read?

Including exogenous variables in your state representation certainly can be useful, as long as you expect them to be relevant information for determining the action to pick. So, state features are not only useful if you expect your agent (through application of actions) to have (partial) influence on those state variables; you just want the state variables themselves to be informative for your next action to take / prediction of expected future rewards.

However, if you only have exogenous variables, i.e. if you expect your agent to have no influence whatsoever on what states you'll end up in next... then the full problem definition typically used in RL (Markov decision processes) may be unnecessarily complex, and you may prefer to look into the Multi-Armed Bandits (MAB) problem formulation. If you're already familiar with RL / MDPs, you may think of MAB problems as (sequences of) single-step episodes, where you always just look at the current state and don't care at all about future states (because you expect to have 0 influence on them).

In theory, the RL / MDP framework is more general and is also applicable to those MAB problems, but RL algorithms that support this framework may perform worse than MAB algorithms in practice, because they (informally speaking) still put in effort trying to "learn" how their actions affect future states (a waste of effort when you expect there to be no such influence from the agent).


Loss function for singular object detection What loss function should one use, knowing that input image contains exactly one target object?

I am currently using MSE to predict center of ROI coordinates and it's width and height. All values are relative to image size. I think that such approach does not put enough pressure on fact, that those coordinates are related.

I am aware of existence of algorithms like YOLO or UnitBox, and am just wondering if there might be some shortcut for such particular case.

The terms error and loss are always relative to some ideal of fitness, and it is always against some model, the extension of the concept of curve fitting.  The ideal is the optimized state.

When we are analyzing system behavior and optimizing PROCESS parameters as we do in machine learning, we don't always use the same terms.  Just as people in medicine and alternative medicine speak of wellness, systems that process data instead of food also can be considered either well or dysfunctional.  With machine vision, we optimize the processing of an arbitrary signal, an image with an object depicted in your case, not the parameters of a curve to fit a set of data points.

It is common to wish to judge the wellness of the learning state across a range of sizes.  You cannot normalize the image by re-sizing before you recognize the image boundaries, so you have to design the process (circuit) that must be trained accordingly.

It is not in the error function that this can be done.  There are only two places where it can be done.

The model of the object you are trying to recognize
The process topology you are attempting to train

The standard approach for size-independent recognition involves a sequence in the process being trained, reflected in activation and interconnection choices.

Edge detection
Corner detection
Shape detection
Illumination detection (for higher reliability)

Note that edge detection and corner detection can overlap in learning layers but not necessarily.

The key is obviously between corner and shape detection.  The feed of corners into the layer that detects shape has to be angular rather than Cartesian yielding this sequence.

Edge detection
Corner detection
Angles at corners
Shape/form detection
Illumination detection (for higher reliability)

A designer can hope a deep network will learn these through back-propagation, but convergence upon the ideal behavior (wellness) will be faster and more reliable if the layers are designed to perform those functions.  That means your model for the shape or form you hope to detect must be in terms of angles not coordinates.

Florian Pierre and Joseph Piewak seem to be doing it correctly: https://arxiv.org/pdf/1709.03138.pdf. Search the document with ctrl-f for "angl".  These researchers show excellence in their treatment: https://arxiv.org/pdf/1806.07996.pdf [2]. Search for "size indep" in that one.

References

[1] Fully Convolutional Neural Networks for Dynamic Object Detection in Grid Maps; Master’s Thesis of Florian Pierre, Joseph Piewak; Department of Computer Science, Institute for Anthropomatics and FZI Research Center for Information Technology respectively; 2016

[2] Novel Convolution Kernels for Computer Vision and Shape Analysis based on Electromagnetism; Beaini, Achiche, Cio, Raison; 2018; Polytechnique Montreal, 2900 Edouard Montpetit Blvd, Montreal, H3T 1J4, Canada


In a neural network given partial inputs and complete outputs, is it possible to predict remainig inputs In example, if there is a simple feed-forward neural network with 3 input neurons, 3 hidden neurons, and one output neuron; is it possible to predict a the value of an input neuron given the values and weights for the other two inputs and the output.

It's important to understand that though neural networks generalize to the whole input space, usually the meaningful input space, from which the training data is taken, is a manifold inside that space.

For example, the image classifier can take any image, including white noise, but it's usually trained on photos, which generally have some nice structure and statistical properties, e.g. smoothness of color, and is expected to work well on photos, not everywhere.

So your question can be reformulated like this: can we learn the data manifold given the neural network and some partial data?

If we ignore deliberately crafted small examples (like ones in which the output just equals to one of features) and consider real world, high-dimensional tasks, we see that the manifold is quite resistant to small perturbations. In high-dimensional spaces a change of one dimension doesn't drastically change the distance to the manifold, it's still very very close. Example: a change of a single pixel is very unlikely to affect the output, hence it would be impossible to predict it. The larger the space is, the less is influence of each individual dimension.

Hope this answers your question.

I'd like to mention here one very promising technique called GAN, which basically does this: it uses one neural network for classification and the other to learn the data manifold. And it works so good that it'd be fair to say that both these networks actually complement each other, not derive from one another.


How close we are to replacing guide dogs with AI? Were there any successful attempts to replace poor guide dogs used for blind people with AI to achieve similar rate of success? I guess dogs could be easily distracted and not reliable for every situation, and it probably takes less time to train AI, than a dog.

Chieko Asakawa (wiki, TED, IBM) is a major researcher in this area, and the linked TED talk is probably a good introduction to the state of the art as of 2015. Here's a link to a 2016 paper on a smartphone navigation system.

Guide animals perform manipulation tasks as well as identification tasks, and so it's not clear if those could be replaced well at all. (A smartphone that reads a label is a great help, but a dog that knows which bottle to grab and deliver to you is probably a much better help.)


Do I have to crossover my node genes in NEAT, and how? I'm currently trying to code the NEAT algorithm by myself, but I got stuck with two questions. Here they are:
What happens if during crossover a node is removed (or disabled) and there's a connection that was previously connected to that specific node? Because, in that case, some connections are no longer useful. Do I keep the useless connections or do I prevent this from happening? Or maybe I'm missing something?
Someone on AI SE said that:

You could:
1.) Use only the connection genes in crossover, and derive your node genes from the connection genes
2.) Test if every node is in use, and delete the ones that are not

But the problem with that is that my genomes will lose some complexity. Maybe I can use the nodes during crossover, and then disable the connections that were using this node. That way, I'm keeping the genotype complex, but the phenotype is still working.
Is there another way to workaround this problem or this is the best way?

Okay, I'm first going to review how NEAT works. I hope this helps you model NEAT successfully as a whole, not just limited to your question.

We use neuro-evolution to create a specific behavior that solves a given task. The behavior can be simple and complex.

Now let's focus on behavior... Different neural networks can create the same behavior (A.K.A the competing conventions problem).

We also want to end up with a neural network that is very efficient.

So we want to solve two problems with one algorithm (NEAT): find the behavior that solves a task and find the most efficient neural network that creates the behavior.

There's a simple way to search for the most efficient neural net: start with the simplest neural net and slowly build the neural net up.

The hard part is defining behavior (how do we define behavior in neural networks?). NEAT introduces their very interesting definition of behavior: an atomic unit of neural network behavior is the connection of the neural network: which node connects to which and with what weight.

Now I'm approaching your question(concern): you want to make sure your genomes can grow complex, which means that you want to preserve the capacity for your genome to express complex behavior. As the paper states, behavior does not have anything to do with nodes, let alone node genes. Behavior is a set of neural network connections.

So here's my answer to your question: care about crossing over connections, not for the node genes. Find an efficient way to calculate the node genes. For instance, you'll consume a lot of memory if you just concatenate node genes from two parents without overlaps. There is a chance that some of the node genes might not be in use (i.e. disabled connections, not inheriting some connection genes due to probability).

Hope this helps :)


Are there any better visual models for transfer rather than ImageNet? Similar to the recent pushes in Pretrained Language Models (BERT, GPT2, XLNet) I was wondering if such a thrust exists in Computer Vision?

From my understanding, it seems the community has converged and settled for ImageNet trained classifiers as the "Pretrained Visual Model". But relative to the data we have access too, shouldn't there exist something stronger? Also, classification as a sole task has its own constrictions on domain transfer (based on the assumption of how these loss manifolds are).

Are there any better visual models for transfer rather than ImageNet successes? If no, why? Is it because of the domains fluidity in shape, resolution, etc., in comparison to text?

Why is ImageNet so popular for transfer learning?

Models pre-trained on the ImageNet datasets have been the de-facto choice for many years now. Many popular reasons as to why people think that ImageNet is so effective for transfer learning are the following:

ImageNet is a truly large-scale dataset that contains over 1 million images, each of which has a decent resolution.
ImageNet has a wide and diverse set of classes (1000 in total) ranging from animals to humans, cars, trees, etc. Most Computer Vision tasks operate in similar domains, however there are some notable exceptions (e.g. medical images).
For example, an object detection model for autonomous driving would benefit from ImageNet transfer learning, as the pre-trained model has seen images with similar content (e.g. roads, people, cars, street signs), even though it tries to solve a different task (i.e. object detection not classification).
The above two reasons allow models trained on ImageNet to identify and extract very generic features, especially in their initial layers, that can be effectively re-used.
ImageNet has a lot of similar classes. This is an interesting argument because it contradicts the second one. Due to the closeness of some classes (e.g. multiple breeds of cats), networks learn to extract more fine-grained features.

Another overlooked reason I find very important is that:

ImageNet has been the benchmark for performance for image classifiers for years now. When, for example, you are selecting a re-trained ResNet to use, you know that that model is guaranteed to operate at a high level of performance. Other datasets don't have such notable challenges as the ILVRC. That challenge is what make the VGG and ResNet popular in the first case, so it comes natural that people would want to use those weights.

In practice, due to the way in which CNNs identify and extract features from images, they can easily be "transferred" from task to task.

Is it actually better than other datasets?

This question was widely explored by Huh et al., who tried to identify the reasons that made the ImageNet dataset better than other ones for transfer learning.

In short they found out that most of the reasons that people thought made ImageNet so good (i.e. the ones I mentioned above) weren't necessarily correct. Furthermore, the amount and diversity of images and classes required to effectively train a CNN has been highly overestimated. So there is no particular reason people should choose this specific dataset.

Now, to answer your questions:

  I was wondering if such a thrust exists in Computer Vision?

No, ImageNet is currently established as the de-facto choice, evident by the fact that all 10 keras.applications models offer weights only for ImageNet.

  But relative to the data we have access too, shouldn't there exist something stronger?

This is an interesting question, as the consensus things that deep learning models keep getting better with more data. There is, however, evidence that indicates otherwise (i.e. that CNN models con't have as much capacity as we thought). You can read the aforementioned study for more details. In any case, this is still an open research question.

Even if models could get better, though, with more data, it is possible that it still wouldn't matter because ImageNet pre-trained models are strong enough.

  classification as a sole task has its own constrictions on domain transfer

There have been numerous cases where models initialized from pre-trained ImageNet weights have done well in settings other than classification (e.g. regression, object detection). I'd argue that initialization from ImageNet is almost always better than random initialization.

  Are there any better visual models for transfer rather than ImageNet successes? If no, why? Is it because of the domains fluidity in shape, resolution, etc., in comparison to text?

Partly, yes. I think that in comparison to text, images have some useful properties that are exploited through CNNs, which makes their knowledge more transferable. This claim, however, is based on intuition; I can't back this up somehow.


Can AI be used to reverse engineer a black box? A while back I posted on the Reverse Engineering site about an audio DSP system whose designer had passed away and whose manufacturer no longer had source code (but the question was deleted). Basically, the audio filter settings are passed from a Windows program to the DSP device presumably as coefficients and then generic descriptions of those filters (boost/cut, frequency and bandwidth) are passed back from the box to the software - but only if it somehow recognizes the filter setting.

I want to be able to generate the filter settings separately from the manufacturer software, so I need to know how they are calculated. I've not been able to deduce how this is structured from observing the USB communication that I've gathered. So, I wonder if AI could do this.

How would I go about creating an AI to send commands to the box (I know how to communicate with the box and have a framework for how these types of commands are phrased) and then look at the responses to either further decode the system and/or create an algorithm for creating filters?

The communication with the DSP mixer box is basically via "Serial" commands and although it uses a USB port, there is a significant bottleneck inside the command control system in the mixer box. Any attempts to reverse engineer may encounter problems based on the sheer amount of time that it would take to compile enough data. Or not.

Yes this is entirely possible. As was previously mentioned, complex connectionist systems are often thought of as black boxes(despite us being able to "look in" the box given enough computation and analysis) because of the difficulty in understanding learning and the networks ultimate decision making.

Here, we can model the problem as such: given an input of filter settings(and presumably some information about the audio), predict the target descriptors as an output. All you really need to do is generate a dataset from the program and then train it in a multi-label classification context to predict the output descriptors.


What determines the values of weights in a neural network? I am trying to understand how weights are actually gotten. What is generating them in a neural network? What is the algorithm that gives them certain values?

Typically, weights are randomly initialized. Then, as the model is optimized for its given task, those weights are steadily made "better" as determined by the network's loss function. This is also referred to as "training" the neural network.

By far the most popular way of updating weights in a neural net is the backpropagation algorithm, most simply with stochastic gradient descent (SGD). Essentially, the algorithm determines how much each individual weight contributed to the network's loss. It then updates that weight in the direction that would reduce the loss.

I recommend going through Michael Nielsen's online book to learn the basics.


Flatten image using Neural network and matrix transpose I have read a lecture note of Prof. Andrew Ng. There was something about data normalization like how can we flatten an image of (64x64x3) into a (64x64x3)*x1 vector. After that there is pictorial representation of flatten

As per the picture height, length and width of the picture is 64 , 64, 3. I think nx is a row vector which is then transpose to a column vector. If there is 3 pictures I think nx contains {64,64,3,64,64,3,64,64,3}. Am I right?
To use a 64x64x3 image as an input to our neuron, we need to flatten the image into a (64x64x3)x1 vector. And to make Wᵀx + b output a single value z, we need W to be a (64x64x3)x1 vector: (dimension of input)x(dimension of output), and b to be a single value. With N number of images, we can make a matrix X of shape (64x64x3)xN. WᵀX + b outputs Z of shape 1xN containing z’s for every single sample, and by passing Z through a sigmoid function we get final ŷ of shape 1xN that contains predictions for every single sample. We do not have to explicitly create a b of 1xN with the same value copied N times, thanks to Python broadcasting.
As per my understanding, Wᵀ = nx and x= nxᵀ.
Is it Wᵀ= [64,64,3,64,64,3,64,64,3] and x = [64,64,3,64,64,3,64,64,3]ᵀ?
In that case there product will be a symmetry matrix.
Is there any significance of symmetry matrix?
I just messed up all the things while flatten the image. If anyone has any idea please share with me.
Thank you in advance.

Yes, if you have 3 images (and by images I assume you mean samples) the flatten layer will be of the shape $12288*3$ ($64*64*3=12288$). The size of $W$ however does not change, and nor does the size of $b$ as these are parameters and are independent of the amount of samples passed through the network.
ETA: I only answered the "Am I right?" part of your question because that's the only part of your questions that's actually a question. I don't know what you're trying to ask in the second half of your question


How can I cluster this data frame with several features and observations? How can I cluster the data frame below with several features and observations? And how would I go about determining the quality of those clusters? Is k-NN appropriate for this?

id     Name             Gender   Dob    Age  Address
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
1   MUHAMMAD JALIL      Male    1987    33   Chittagong
2   MUHAMMAD JALIL      Female  1996    24   Rangpur
2   MRS. JEBA           Female  1996    24   Rangpur
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MR. A. JALIL        Male    1987    33   Sirajganj
3   MD. A. JALIL        Male    1987    33   Sirajganj
4   MISS. JEBA          Female  1996    24   Rangpur
4   PROF. JEBA          Female  1996    24   Rangpur
1   MD. A. JALIL        Male    1987    33   Chittagong
1   MUHAMMAD A. JALIL   Male    1987    33   Chittagong

A typical clustering algorithm is k-means (and not k-NN, i.e. k-nearest neighbours, which is primarily used for classification). There are other clustering algorithms, such as hierarchical clustering algorithms. sklearn provides functions that implement k-means (and an example), hierarchical clustering algorithms, and other clustering algorithms.

To assess the quality of the produced clusters, you could use the silhouette method (sklearn provides a function that can be used to compute the silhouette score).

Regarding your specific data frame, note that it contains repetitions, so you may want to remove them before starting the clustering procedure. Also, the IDs are not unique, but you probably don't need the IDs for clustering.


Has anyone thought about making a neural network ask questions, instead of only answering them? Most of the people is trying to answer question with a neural network. However, has anyone came up with some thoughts about how to make neural network ask questions, instead of answer questions? For example, if a CNN can decide which category an object belongs to, than can it ask some question to help the the classification?

Maybe neural networks are not the best tool for this.

It seems to me that an equivalent of the your notion of 'a question to help the classification' would be to use Machine Learning (ML) to obtain a human-readable ruleset which performs the classification. The idea is that, if you follow an applicable chain of rules all the way through to the end, you have a classifier, if you stop before that, you have an indicator of which features of the input give more coarse-grained classifications, which can be seen as a progressively detailed sequence of questions that 'help the classification'.

More detail on various options for using ML to create rulesets can be found in my answer to this question.